I found PE classes daunting as soon as the team games were introduced, because I could never understand the rules of the game. So I'd just absent-mindedly do what I thought was right then let the team down in such humiliating ways. I remember one sports day when I was about 9, I was doing one of those obstacle courses, with lots of parents watching, and no matter how hard I tried, everyone else ran ahead of me and through the obstacles, while I was miles behind, feeling hopeless. I think it was the anxiety of being watched by lots of children and parents, as normally I quite enjoyed obstacle courses.
After that, my mum always let me have sports day off school because she knew how anxious it made me. I don't think children should be forced to participate in team games. They should be extracurricular activities. In mandatory PE classes children should be doing real exercises and games that don't involve harsh competition.