The caffeine makes me feel very irritable and unpleasant, this alone can make me feel sick from the horrible feeling of anxiety it can cause, especially in strong coffee. I also have a lactose intolerance which makes me start to feel sickly, sometimes bloated, often with stomach cramps if I have more than a very small amount of normal milk, if I have a lot of milk or other dairy products I can actually be sick, but I now know to avoid this and lactose intolerance is very common.
There is a solution to both of the above issues however. Firstly there's obviously decaffeinated coffee which many people seem to hate the idea of, but I've found that if I buy a quality name it's surprising good and it can be difficult to tell the difference, they seem to have come a long way these days with decaf coffee than in the past when it was noticeably inferior in taste. Secondly lactose free milk has become readily available in mainstream supermarkets and it only costs slightly more than normal milk, plus in my opinion I really can't tell any difference in the taste and I never buy normal milk any more. You could even use soy milk, I personally like it sometimes, but it does have a different taste which you may or may not like, also I've never tried it with coffee and some people complain that it causes it to curdle, although this can apparently be avoided if you warm the soy milk slowly by pouring it into the cup first, then gradually add the coffee. There are other alternatives too including coconut milk, almond milk, cashew milk and more which some people love, but again they have all have a different taste. Vegans use soy milk and/or other milk alternatives all the time for everything, I've been vegetarian since 1996 so I can have dairy products at the moment, but I keep them to a minimum and I'm bordering on going all the way.