You are. Youre worry that he's not attractive from far away??? Are you kidding? Also, just because his brother may have something wrong with him, so what? You're not going to find someone who is perfect AND everyone related to them is perfect too.
If you're worried about what your kids will look like, or that he's not attractive enough? Some of the hottest guys I know have the ugliest parents, and vice versa. Also, might I add, that you both could be completely healthy and have a kid who has a disability or health problem.