As you know, I don't lack assertiveness, so it wouldn't be a problem for me.
But, on reflection, if a person isn't confident in approaching a person in such a situation, perhaps changing times might actually be the better option if there is no inconvenience.
Approaching management without talking to the guy might punish someone who may not deserve it.
I have been on the receiving end of many attempts at character assassination myself, hence my interest in this thread. I'd rather an innocent person isn't unjustifiably vilified.
The guy may be a creep, but we don't really know.
My policy is to avoid conflicts wherever I can. However, my assertiveness has paid dividends from time to time.
But, "whatever floats one's boat".