Who are these people around you that think it is "sick"? Are they close friends? Family?
Or are they colleagues or acquaintances?
There are many reasons why people might try to make you feel bad. If people see that you are happy and in love, it can be a cause for jealousy. Specially if their own life is crappy.
If they don't find this to talk about, they will probably find something else to criticize, either about you or about something else. People are bored and things like this which are a little bit different, serve to make their boring existence a little more interesting.
You are 23, he is 40, both adults in a position to decide what you want in your lives. I don't see anything that makes this "sick".
