Inbox, this happens to us all & is so annoying! Next time you arrive home & find that whatever item you purchase is the wrong size or somehow not right, put the item along with your store receipt aside until you can return to the store. You probably don't want to have to bother making another trip to the store or feel ike having to explain the predicament to one of the store clerks, because it is an inconvenience. But unless you went ahead & used or wore the item, or lost the receipt which provides your proof of purchase (& show the price you paid), most stores have no problem with someone returning an item that isn't right for the purchaser (i.e.; the customer, in this case, you).
You don't need to worry or "confront" anyone at the store. Just calmly & nicely explain that the item is the wrong size, you didn't wear it & here is the receipt. They will usually ask whether you want a refund or another item in the correct size. A good store clerk will want to help you, & make sure you are happy & pleased with your purchases & shopping experience. Of course there are jerks in every walk of life including retail store clerks. But they should be the exception versus the norm.
Just keep in mind in advance that some stores have a policy where they provide only a store credit or replacement item and not a full money refund. Most stores also have a time limit on returning an item which for some reason can commonly be 30 days, or 90 days. Most stores will not accept worn clothes for return.
So if this happens again - or your pants purchase was recent! - don't worry about returning them!
I know what it is like to purchase something & get home to find it is not exactly right. I have told my husband many times that I will one day hold a "garage sale" to sell all the items I purchased but never used or wore because they were the wrong size or whatever; and I never made it back to the store in time to return them! Ugh