Can I carry a garment bag, a backpack and a carry-on baggage on the airplane?

You can try. Most airlines allow a carry on (roughly 9" x 22" x 14") and a "personal item" like a purse (some of which are as big as a garment bag) so that's two allowed items. You're probably going to run into problems when you add the third item especially if you are traveling now (near the holidays). That said, if you get on the plane early and the gate/flight crew doesn't have the time or energy to enforce the rules you might just make it. Give it a try and the worst that will happen is that you'll have to turn over one of your bags at the gate and it will be checked and stowed under the plane. Don't expect the other travelers who are following the rules to be very happy about your attempt though...

2 items meant 2 items. Today's airplanes are almost always full. The airlines have to be strict with the guidelines in order to accommodate a full plane of passengers.

if not you can pay extra.

No. Too much.