I am not certain I am aspie, but I struggle with phone calls as well, in the same way that you do. My mind goes blank and it gets harder to find information, to read what the person is saying, and to pick up on social queues. I think its because its a deceptive middle ground between being in person and the written word.
With the written word (email), there is no body language to read, but there is also no pressure to know the answer right now - no sense of immediacy. You have time to analyze and pick apart the communication and determine what the person means and decide on your response. With the phone, you are often missing a huge portion of important queues used to determine what the person means, but they still expect you to know the answer right now. So the brain goes blank because it needs to do at least as much analysis as the written word in the time of a spoken conversation (among other things).
Also, I know that the way I think relates to the creation of spatial data. By this, I mean I do not think in words or pictures as average people do, but in spatial data or I call them 'holograms'. Like memories or experiences. So if I am talking to someone on the phone, I am experiencing a sort of 'hologram' of them talking to me where my brain tries to construct a sense of them on the phone with emotions and expressions ... but then as they talk, my brain also switches over to creating holograms of what they are saying ... because that is how I think ... so I constantly have to switch back and forth between seeing/experiencing what they are saying and seeing/experiencing them as a person on a phone talking to me. Which is not only exhausting, it makes it harder to think creatively/abstractly because I'm already using a lot of my abstract reasoning constructing them as an experience. (note that this isn't a choice, its just what happens. I'm speaking as an observer of my mental process.)
So I try to avoid phone calls as much as I can. However, if I know I'm going to have to talk on the phone, like at work .... I try to write conversation scripts and/or talking points, and important information that I might need before hand so that I have it. The less thinking I have to do when I'm on the phone, the more control I have over the conversation and the smoother it goes. I also have information that is often requested info in easy to get to places... such as a note card attached to the side of my monitor with addresses and numbers on it that I need to give people when I'm on the phone at work.