You feel guilty because of cultural conditioning.
In other cultures, the wives feel relieved that the husband can get his rocks off with a sex worker and come home pleasant and charming.
Men just like to have sex - we are biologically programmed that way. We remain fertile til the day we die...