You take it too seriously. It's common to relate things to feel more comfortable in a situation. I know what you mean, it does get annoying, but think of it as a complement next time. Be happy they think you are on the same caliber as a very famous magician, like David Blaine who is good enough...