I think I never had a date with someone who would not be dressed nicely, then again I believe people in the US dress more casually in general than here in Europe.
For me it's normal to put some effort to look good when going on a date, each man has a different style though so I don't really judge a man's style that much, then again I don't think someone would go on a date wearing gym clothes or something similar here.
A friend of mine had a date with this man who apparently was dressed as if he was about to go hiking, and she didn't like it. I think I would have reacted the same way in this situation.
As for your date, I think he was probably disappointeeld that you decided to leave after 30 minutes so that's why he asked for an explanation.