As long as you are honest in why you do magic, there is nothing wrong with that.
Did you get into magic because you like performing? Did you get into magic to get over social anxiety? Did you get into magic to attract the opposite sex?
If you know why you are in magic, then it doesn't really matter. It makes you a real person. I HATE when a magician is in a conversation in real life and says "Hey Katie, can I show you a trick?" I say "sure" and then he changes into a character. I liked the guy I was chatting with, not that guy that now feels that he has to go into ninja performance mode.
If you do it to help you get a date, that's fine and I applaud you. I'll be honest with you and tell you that a lot of girls don't really care.
If you do magic to gain confidence, thats awesome and I applaud you. And it does help, as long as you are YOU.
Anyway. These are my thoughts on the matter. As long as you know who you are, then people shouldn't be annoyed, they shouldn't care. Just make sure you are honest with yourself.
If you would like to discuss further, you may PM me with any questions.
Thank you,
Katie