Regarding the struggles, difficulties, and stress associated with "not looking at other women". What is actually meant by that? Seriously, who doesn't look at other people? You can't navigate your world without it. Of course, I am going to see women in my environment. What I think is meant by that is more in line with my previous statements about whether or not one can appreciate beauty in another person and not act upon it with lustful behaviors. If you have self-discipline and control, there's no stress, at all. Perhaps one reason why my wife and I have been in the relationship we have been in for so long. It gives you the intellectual freedom to just be yourself and relax with others of the opposite sex without your partner getting concerned.
I am more in line with your thinking that trust and loyalty are important. If you or your partner are distrustful for one reason or another, it would be more difficult. If either of you are bringing that sort of "baggage" to a relationship, then that's a huge hurdle to jump because if there is any stress, it's in the fact that there are often toxic, insecure, controlling behaviors a play. "He shouldn't be looking at other women!" Really? You don't look at other men? Of course people are going to look at other people. Your insecurities are getting the better of you and that distrustful behavior will tear your relationship apart, not his supposed "wandering eye". He hasn't done anything that you don't also do. If there is any stress, it's being the victim of controlling behaviors.
I am very heterosexual with, what I would think, a high sex drive. At work, I joke around when I say, "I swim in the estrogen ocean" amongst a predominantly female staff, roughly 90% I would guess. I have been their listening post for decades, mainly because I am one of those men they are comfortable with and trust enough to ask questions about relationships. I have sat in the break room with a group of women discussing all manner of "racey" things. Literally thousands of women all around me, and yes, a fair percentage are young, beautiful women. Here's the thing though, it's a professional environment and they all know I am married. I simply have to have the self-control and discipline to shut off any impulses and feelings I might have and interact as a friend and co-worker. There's no stress. This narrative that men, in general, cannot control theirselves, that they are weak in this regard, I find rather insulting.
I think some men are weak, for sure, as are some women. Men and women who do not have self-discipline and control, in my mind, are weak individuals. Children who grow up without some degree of Stoic thought from a strong, present father figure to teach them those mental skills, may be more susceptible. Children who grow up who are allowed to filter their actions through their feelings, may be more susceptible. Intellect also plays a role in how much in self-control one has. I would suspect those with a lower intellect probably have a higher emotional bias in their thoughts and actions making them more susceptible.