I can totally relate to that. I used to try a lot harder to make friends by inviting people out for coffee or lunch or something (as a student or when I started working) and the pattern was always that they agreed to meet me once, we had a rather awkward conversation because I struggled to find common ground and maintain it, and they didn't know how to relate to me either, then it would never be repeated. I don't know anything about TV and don't watch all these shows that people seem to follow and talk about, rarely find people who have the same taste in music or literature as me, and don't keep up with movies, I don't have kids or the same sort of life as others my age have, so what would we talk about?
I've since learned that it's much better to plan meetups with others round activities rather than cafes or restaurants, where the principal activity is talking. For example, if you go to see a movie, you watch the movie and don't need to talk, then when it finishes you have something to talk about, the movie. It takes the pressure off talking. Or you could go bowling or play games of some sort, then the focus of the meeting is on the game, not socialising, and you automatically have something to talk about - it really breaks the ice. Or a walk in the forest or in the countryside if you prefer a calmer environment, a trip to a museum.