I know this post will sound wikedly conceited and I might get bashed for it.. but.. maybe this will help.
I was in love with a really ugly guy. Most people consider me average or above average (depending on taste i guess) so I'm not "ugly", but most people would never give him the time of the day. I'm not even exagerating.
He has a slightly unibrow... huge huge lips.. his eyes have permanent bags on them.. his nose is huge and croked.. but again, I was REALLY into him. My own mother (45) lol-ed when she saw him. She literally laughed out loud.
Most people thought it was a joke that i'd even went out with him - let alone fell head over heels for him - but i did and hard. Why? It honestly was because of his personality - or what i thought it was his personality (but thats a whoooole different story).
While he was ugly in the face, he didn't lack "manliness". By the way you talk, maybe it's just this post since you are complaining, but still - you will never caught him dead talking like that - there's a certain agency, he refuses to be victimized and seem as (especially BE) helpless. That's what I loved about him the most. He worked out and he was determined. He had a strong drive, was genuine and caring. All those things made me admire him.
So if jokes are all you have to offer, you really are SOL!
He was my bestfriend for about 5 years before I started to become really attracted to him, though. But I definitely was in love or I wouldn't have been so devastated.
So, to sum this up real quick: as cliche as it sounds, it's true. Personality is really the key to attracting others.