If prostitution is out of the criminal environment, it will become a job much like any other massage therapist. In many European countries it is perfectly legal, and you don't see more abuse or exploitation.
And -perhaps- this is the only way for some women to make real money.
However, that is their job. If I end up without a job, my job options would include different jobs, ranging from burger flipper to gay prostitute. The choice would be mine and mine alone. If I voluntarily choose for the high pay, then noone should feel guilty about that.
Similarly, those women are not facing a choice that others aren't facing either. They choose that job, others won't. The only problems that are actual problems are caused by the criminality of prostitution. Change that and you stop the exploitation as much as possible.
I really invite you to look at countries where it is legal before jumping to conclusions.
Personally I think any single man should be able to visit a prostitue if he so likes. It's called consenting adults, and really no different from a guy 'buying' a girlfriend with gifts, like hugh heffner or the late husband of the late Anna Nicole Smith.
And yes, I am a male.
And I think having sex for pay is really no different from getting a back massage for pay.
Infidelity is morally wrong inasmuch as it is the breaking of a vow you made, but plenty of single men use the services of a prostitute.