I am curious can someone be asexual but still be physically attracted to someone? I am only asking this because I have a guy friend who has never had an intimate relationship. He is in his forties I I am pretty sure he is attracted to me . However at his age and never having a sexual relationship makes me wonder if he has no desire and possibly asexual.
I really think you're looking at this from the wrong angle. It is understandable that you wonder, given he appears to lack prior intimate relationships, and for an NT person that would be extremely unusual (though not impossible) by that age without there being a simple explanation like this.
However, he's not an NT, and it isn't really helpful to attach more typically NT-related behaviors to him as explanations.
Because we're so different in many of the emotional aspects of relationships, we tend to experience rejection very commonly, and often it is increasingly damaging to self confidence and ultimately even our sense of actually being able to engage in a meaningful relationship. When all your get is rejection, a rather natural response is to stop trying.
What would make it harder is that if he can't read your motives and intentions, and many of us can't because they're not simple and direct statements of fact but signs and signals being given off for us to interpret somehow, there may be nothing to tell him how to proceed, or even what you want.
Him spending time with you, even when he might have other things to do, could be the only sign he knows to give you that a relationship and being intimate is what he wants, and the fact you're not responding to that simply compounds his belief this
isn't what you want.
I'm not in favor of you simply asking him, because firstly that creates a problem he may not have confidence to answer, and it's more likely he'd avoid a response or provide a negative one than anything, in order to protect himself from more rejection, but you likely will need to take the lead rather than being passive.
It is also essential that you realize that he may even identify as asexual, since that would help him explain the lack of intimate prior relationships, but like other sexualities, it can be highly fluid, not fixed like a block of concrete.
While possible that he doesn't find you sexually attractive, there are a million other far smaller things that can be in the way - not many of which are amenable to him explaining. Fear is by far the most likely.