It is good he contacted you back, but this is a tough situation to know what to do, as his timing desires and needs seems to be different than yours. If one of you does not bend in terms of timing, the risk is there that one will get offended or upset.
It is like telephone tag in a way. A person calls another, but they are not there so they leave a message. The other calls back, but then that other is not there so they leave a message saying they got the message. And these messages continue until each can reach one another, increasing the frustration.
In other words, if a person acts either too evasive, busy or wants things done just at their timing, that does not show a priority or empathy, whichever is the case. In your case, if chemotherapy is involved, I would say you have a more urgent need for a delay to meet, if the visit is a longer visit.
But, if the visit could be a shorter one, over the weekend, if you felt well enough, then that is an option, too. If, however, your health could be affected by a meeting sooner than you desire, of course just tell that to that friend. If he cannot accept that, then that is not a good sign for you both.