The truth is, it does matter and women do take notice (I actually wrote an embarrassing post about a comment from an ex gf yrs ago). But, it doesn't matter more--and probably less-- than other things.
After all, guys have physical preferences for women. But by and large, they aren't deal breakers.
I once had a girlfriend-- an ex, years ago-- tell me that I was small. Don't think she meant to offend, but I didn't love it. Funny thing was, when I mentioned it to her much later, she thought it was funny, but hasn't remembered saying it.