Talk to him about it. Tell him why you feel the way you feel, and what you are worried about. If he truly cares, he will respect your decisions whatever they may be. What you said in your post about your feelings is probably something he needs to hear.
For immature guys, sex is a status issue - but I don't really understand why. There's nothing that needs to be 'proved' if there are sincere feelings and love in the relationship.