You have to remind yourself that even though you're a great performer, if a heckler wants to mess your magic up, he will. Depends on How determined he is.
I once performed randomly for two ghetto guys who were quite aggressive when it comes to magic. They heckled like mad, but i carried on...