There are two issues here. The first is a matter of control and the second is a matter of challenge.
The matter of control is that you most likely are not acting as if you are in control. They way you should present magic is to demonstrate that you are in control. If you are nervous or of you telegraph when you are making a move or when you are "dirty" - the audience will notice. If you are confident and in control, spectators are less likely to interrupt or try to take control.
The matter of "challenge" was touched on by
@DavidL11229. Chances are if you are performing "tricks", the spectators feel like they have a choice between being "tricked" or "fooled" (and nobody wants to be a fool) or figuring out the method. Now, spectators are often conditioned to do this... Fool Us, the Masked Magician, etc. reinforce this silly game. For younger magicians performing for their friends, this is even worse... magic is viewed by the performer as something that enhances their status while the spectators try and figure it out to knock them down a peg. The way out of this is to focus on having your spectators enjoy the mystery. Make your performances something special rather than "hey, you want to see a trick I just learned?"
Finally, as the other David pointed out, there could be issues with the structure of the effect if they have a reason (other than being an obnoxious prat) to think seeing the other side is important.