I am not married, but I know what you are going through (as I am sure a lot of us do!) Sadly, most people who have no experience with age gap relationships just don't understand. You can't help who you fall in love with! I am not married, but I am in a relationship where we have twenty-one years between us. We have a few supporters, but most people who know about us are very vocal about their opinons.
I can relate to what you said about the chatter making you depressed. Sometimes it makes me really doubt my relationship with my significant other, but when I am around him all those doubts go away. So I know that it's right for right now. If you get the same feeling, then I don't think you should worry about what they are saying.
