I'd definitely agree with "just let him do it".
Though it's very hard to explain why. Sensory issues are the sort of thing where those who have it, fully understand it, but those who dont, probably cant, because there's no way to REALLY explain what it's like.
I have plenty of sensory issues myself (though not that one) and anyone trying to take something from me is, well... it's not a good idea, I'll put it that way.
It's the same with the idea of object attachment, which is different than sensory issues. For instance I have this keychain, right. It's like 2.5 feet long (no, seriously). I must have it with me when I'm out of the house or it'll just bug me the whole time. But more than that, I dont let anyone else touch it if I can avoid it. If someone so much as picks it up just to move it out of the way, my full attention is focused on them until they let it go. No, I dont know why. But object attachment can be very intense. Well, many things related to the spectrum can be "intense".
In a general sense, for someone on the spectrum, taking something like that away is often a very fast path to big trouble. If you've heard the term "meltdowns", that's one thing that can easily happen. As long as the thing he's attached to (regardless of the reason for it) isnt harmful, it's best to just leave it be.