It sounds like he is blaming you for his feelings of worthlessness. That's where he's wrong. He should have told you earlier on how you felt, so you could change before. He needs counseling in a bad way. I would get out of the relationship and suggest he seek counseling, and you do the same for yourself to learn better relationship techniques. I did that after I broke up with someone, tried to go back and he decided he didn't think he wanted to try to make it work. You can only do your own work. This guy needs to take responsibility for healing himself. you cannot do it for him.
Best of luck,
Belle