The guy will probably cheat on you too. For me, this person is not worthed. If you're unhappy with your wife, devorce her, and after that have all the freedom you want and may be you'll find the right one. Cheating while married shows a lack of responsibility and values(the ones that will help you create a new happy family). Sorry to say it but I think that you're kind of the same person like him. The fact is that fears like the ones you have will never go away and your relationship is doomed. Think about it, explain that to the guy, get your head straight and find somebody else to live a happy and trusted lifetime with. That's my honest opinion based on the fact that he's done it with many other women before you. Good luck and I hope that everything works out for you.