Hey bro, don't be so hard on yourself. From your post, your heart is clearly in the right place.
The race you're running may be more marathon than sprint, so steal yourself for the long haul. It's completely understandable that your wife is pretty down about the infertility and it's understandable that you need a break from being down. I think you're there already, but while you're forgiving yourself throw an extra portion of forgiveness her way.
Your role as husband is to love your wife. Maybe that means just listening, maybe that means giving some tough-love-perspective. Lord knows I get wrong plenty with my wife. The point is to try and remember the ultimate objective when you get home--to love her. You go into your conversations with that perspective you'll be able to hit the mark more often. Prayers bro.