This is almost exactly how I feel. For me it's a lose-lose, and I see this as a ruined weekend. It's going to occupy my brain for the next 72 hours. I do kind of feel bad about not going. On the other hand, I'm angry that someone is so insistent that I go to something. And I feel annoyed that they actually think I want to attend and want to interact with all the employees' spouses. What they think is doing me a favor is actually a huge inconvenience, and if they listened, they would know that. It just seems so presumptous.
I totally understand your frustration, but sociable people assume all people enjoy socializing. In other words, they just don't get it. I know this advice is strange but since this has already ruined your weekend, how much worse can it be if you go? You said that the other coworkers seemed obliged to go as well, so at least you won't be alone in that regard. Honestly, I would feel less comfortable if I had coworkers of a similar age, instead of older ones.
With older people you can blend into the background much easier and since there's an age gap, they probably won't be offended. They might even feel a bit sorry for you if you're the odd one out, and if you didn't enjoy it, they may let you off the hook next time.
Since this coworker is so insistent probably means that if you don't go, the issue will drag on beyond the weekend. Rather nip it in the bud, and get it over with. I know it's difficult as I've been there before but sometimes as I've experienced getting it over with is the lesser of the two evils, if you like.