Whenever someone tries to goad me by saying "you're this and that" I just say "you're probably right" and go on with my day. It's difficult for people to argue with someone who agrees with them.

You don't "have" to respond any other way.