I was about 19 I guess and in Las Vegas. I'd just done a pick-pocket routine and was back stage when my friend Peter pulled me over to meet a certain gent by the name of Liberace. As I reached my hand out to shake his he suddenly drew it back exclaiming, "I just saw your act, there's more than a million dollars in diamonds on that hand, you think I'm a fool?!"
Of course he then started laughing and ended up giving me a hug and compliments, but more than anything he gave me a wonderful memory.
Even when I was in the Navy there was the joke that I wasn't allowed to play poker with the guys because I was too good at doing magic (little did they know how bad I was when it came to card tricks).
The point is, this is a very common pun people have when they know you do magic. I've put up with it since the early 1970s at least. But, there's a very interesting thing to all of this; when I shifted away from magic altogether and focused only on Mentalism, that kind of teasing was replaced with psychic type gags... "What's the weather going to be tomorrow?" "Which team is going to win the ___ Bowl?" and so on.
I really doubt that there's a trade that doesn't have such silliness tied to it, as show people however, we're just more susceptible to it.