Imagine a kid at school that ALWAYS carries around a yo-yo. Whenever you see him, there he is with a yo-yo in hand doing tricks. Not only is he doing tricks, he is asking people if they want to see his yo-yo skills. Some people say yes, because they are his friends, and some people say yes because the kid can actually do some pretty nice tricks. Regardless however, no matter what, he always has a yo-yo out. It seems quite obvious that this kid would start to be known as the "yo-yo kid." Not many people around will actually stop to think about "why does he always have a yo-yo?" they just know that he always DOES have a yo-yo. And, since the yo-yo kid always tries to show kids his tricks, some people will think it is ok to ask him to perform those tricks for them too. When the kid refuses, they might be confused... because "wth?? he always shows tricks, and now when I ask, he won't? WTF?"
Do you see where I am going with this? You have made yourself into the magic version of yo-yo kid. You are not Rob, the cool kid that is actually also quite a good magician, you are the magic-kid, whose name happens to be Rob. Your intentions might have been harmless and you may have started out doing it to try and boost your self-esteem, but magic has become your crutch to society, to your sense of belonging, to your mental well-being... none of these is a good thing, in fact, it is an unhealthy thing. Imagine you were to permanently damage your hand... I get the feeling that you would feel your life to be pretty much over. Sure, it would be a disappointment for me to for example, lose a hand, lose an arm, break a hand for a few months, etc... I would not be able to play pool, do magic, play the guitar, etc... but I have other things around me to live for. I really suggest that you take the time to figure out what else you have going for you because I am sure you have plenty. You really just need to develop your social/ communication skills outside the realm of "performance."
Learn to communicate with others. Educate yourself. Work out and exercise. Do volunteer work. You said that you do magic because you have low self-esteem, and it gives you a sense of belonging, and that people genuinely appreciate you for it. To a point, that is all well and good, but if your motives for magic are 100% about yourself, you are not going to be able to dig yourself out of these situations that happened at the bowling alley.
A normal person would have been upset at the fact the other person was being a jerk, but nothing more. You are upset at the fact that your magic wasn't appreciated, that you didn't know how to handle the situation, and that you don't know when you should do magic or how to turn people down. That is way more than you need to be worrying about right now. And, I feel that it is your magic that is causing these problems.
Gotta have balance in life my friend. Or you will tip over and not be able to pick yourself up.