make everyone happy ... masturbate beforehand, and fake it when you ... it's not the best advice, but at least you wont hurt his feelings ... or, you could blunt about it and just tell him.
and if this is going to be a major problem for you, then break up with him and find some bloke who fits your standards of Big ... and if you break up with him, think of some other excuse, dont tell him he is 'small' because you will shatter his confidence with other women.