I'd imagine that most people don't like other people yelling at them, myself included. When I'm not rambling about things I'm fixated on to people I'm usually very quiet, so I get very used to the tranquility (in school and other loud places I usually have music playing in my headphones to block out the noise, or in the first place I'm paying close attention to it for whatever reason, usually out of paranoia). When big things happen it really shakes me up inside, and I'm left reeling from it all as if someone actually hit me. When my mum and stepdad have fights there's usually banging and the whole building feels like it'll all fall down from the vibrations; that leaves me deeply unsettled, shaking from the head down, desperately hoping that it'll all go back to the quiet it used to be. I'm left wanting to curl up tight in a nest made of blankets, or doing my favorite things on the computers just to calm down enough to sleep (I think physical affection from a deeply trusted female source also works, as I often want someone to cuddle with during those times, and cuddling with a male is deeply disturbing for me, unless perhaps if they were any future children of mine. The list of deeply trusted people only has my mum on the list though, so no dice there).
When people yell at me when I have the even the slightest hint that they could in anyway be yelling at me it depends; if they were yelling to get my attention it doesn't really do much besides the initial shock, as by then I'm already either aware it's not me and put on my headset again, or I'm already on my way to find out what they need; if they were yelling at me with even the slightest hint of aggression or anger I'm almost utterly unable to speak (any attempts to just leave me stuttering, which is frustrating and only adds to the stress), but on high alert for any aggression. I can't remember exactly how I reacted during any times it felt like it would turn physical, but the affair always leaves me jittery and ready for violence even in the haze of it all; if I felt stabbing someone would keep me safe I'd probably do it if left alone with them (pretty sure I was all about to do that with one of my bullies if I didn't get convinced to leave for home).
When it's someone I actually care about yelling at me (So far the list has my closest cousins, my mum, and my brother), whether in disappointment or anger, I'm in quite the quandary. I can't retaliate because I actually care about that person, but I can't walk away both because I'm too shake up and because it feels like delaying the inevitable (I'll have to talk it out with them if I ever want to go back to normality).
I must admit that next to all of your full-blown panic attacks I feel out of place. Regardless, I hope that life would be kind to us all.