What you ask is really a situational thing. I know you already provided your situaton but I'll just touch up on what I think.
I don't perform a set audience for a time often as I'm not a professional. Done it a few times but yeah... will talk about that later.
I see fillers as things to continue the wave of magic. You don't want dead air between tricks/ routines. Although it would work to follow routine with routine, hitting your audience square in the face with masterpiece after masterpiece, I think that might exhaust them a bit. It's like watching a movie with explosions after explosions. People think "holy crap thats amazing!" way too many times that they just find it overwhelming.
It is true that most tricks can become deadly if you give them the right time and work into the presentation but you don't want all of them to smash your audience. Performing should be like a work out. It starts out slow, builds up, reaches a peak, rest for a bit, reach a peak, rest for a bit, reach a peak, (throw away your cool down and end with a bang).
When I read your first post, I immediatly thought of Malone. He uses comedy and short sleights as fillers. He has just finished a routine/ trick. Then he jokes, laughs, socializes, and once in a while he coughs up some cards. People acknowledge the end of a routine, they relax, they take in what they just saw and want to let their mind get ready for the next thing. If you show them trick after trick, they will probably forget most of what you perform earlier and it all just becomes one big mess of awesome.
"how was the show?"
"awesome"
"whatd he do?"
"i dont know but it was awesome"
Fillers allow their mind to comprehend each section/part of your routine, allowing it to settle into their minds and lock in. They will remember each part clearly and know how your show progressed.
The fillers continue the wave by not stopping totally. You dont want to be performing card magic and lets say some tricks require a setup or some other "four of a kind". Fillers are just words, patter, and maybe small effects which fill up the dead space
You don't want
"and there is your selection!"
"oh wow"
"wait a second" *rummage rummage*
"now pick a card"
You want it to flow.
"and there is your selection!"
"oh wow"
"now have i told you about the...." *rummaging* *little flourish over here, little production over there* "..and thats what she said."
"hahaha!"
"and now will you pick a card"
Fillers are not to be used as another trick though. If you perform an amazing trick and suddenly cool down with something less impressive, your audience will definitely just think about that last one and talk about that one, deeming what you just shown them as "cool". If you performed a killer invisible deck, you wouldn't really want to follow that up with a inversion plot would you? Fillers are just there to continue your flow of amazingness until the next big thing.
For example you perform a killer card trick and want to move onto the next killer. You may finish and start shuffling cards, "accidentally" shuffling them face up and face down once without noticing. Your audience notice, they tell you, you show that nothing is wrong, no biggie. In their minds, they think "what the hell did I just see? that was weird..." but they dont think its one of your tricks, so its not necessarily a "oh.. thought he would do something better..." They will still remember it but will remember it as an amazing small moment instead of a let down after a mind rape.
As for non show/ set performances as in for friends and what not. I would just end with the biggest thing I have at that point. Like... what could you possibly follow up with after you perform that trick which is pure luck? "shuffle, and pick a random card, now you there name a random card, it matches? great!" What would possibly top that? I have successfully performed that 8 times in total and have stopped immediatly after it. Anyting after it would just be small beans. If you are performing a set show or anything, I would just stay away from oppurtunities like that. Sure the payback is amazing when it works, but what if it works mid show. The second half of your show will just look weak in comparison. You could try it right before your finale but why would you want to cut out your finale? its called a finale for a reason.