This is a three part problem you're trying to deal with here, and unless you are able to address all three parts of it, you're not going to get the exact results you desire. This is one part audience control, one part presentation, and one part character.
Here's the breakdown. Your patter needs to be involving. If you are not activly doing something "magical" then your set up needs to have motivation and meaning. Without knowing your presentation I can't say that you aren't already doing this, but it certainly is something that you should stay focused on.
Audience control is also a must. Don't be afraid to flat out tell people, "Lets stick to what we're good at. I'll do the magic, and you just watch." Don't be afraid to slow the show down, slow the audience down and take it at a managble pace. If they don't want to do that then simply stop the performance. The one thing worse than a poorly done trick that exposes the method is a well done trick which is rushed and the meaning and magic is lost to instant gratisfication.
Lastly is character. You have to give them a reason to invest in you. Give them a reason to want to engage you. If you are doing a trick for the sake of doing a trick then you are openly challenging them to watch and try to figure out a puzzle. Give them something to connect to you on and you will find that they'll follow you to just about anywhere.
I'd add something else to that.
It seems to me, from what you've mentioned about your presentations, that your presentations just aren't convincing.
Honestly, this is a pet peeve of mine. Take for example, the Time Machine trick. First sentence:
"This Ace is a time machine."
In that one sentence, you've just lost your audience. Congratulations.
Presentations must must must make sense. No-one in their right minds will believe that the Ace of Spades in your hand is a time machine. Don't get me wrong - the presentation doesn't have to be exactly what's happening - they don't have to believe every single word of it - but it has to involve a logical enough leap so that it makes some sort of sense. Even if you take a story presentation - at least a story makes sense and explains what's happening.
But these guys have seen playing cards before. They know that Jokers aren't actually wild cards. They know that if they hold them, they won't change. So they don't care about the presentation, they just want to see the change. They know you're not invisibly transporting cards from one hand to another. The card is not invisible. When that happens, they stop caring about the effect, and latch onto the only thing they CAN care about: the expected moment of magic.
That I think would be my explanation for what has been happening to you.
Quite honestly, if someone walked up to you and explained that the ground underneath you could be a magic carpet - how would you feel?
Irrelevant patter created solely for the sake of having patter has two effects: Firstly, it appears contrived. Which destroys the illusion of magic - they just wanna see "the cool part". Secondly, it actually insults them - it's an incredibly patronising thing, to have someone explain that your card is a time machine, and it's an insult to their intelligence.
The only way that presentation exactly works, is if you perform it tongue in cheek - à la Gregory Wilson. But that's a different story altogether.
Anyway, the point is this - your presentations have to make sense, or they'll be dismissed, and the magic will be gone - because, in reality, to be completely truthful, your patter will in fact be contrived and patronising.