So here is my question, when you improvise is your patter more than "when I do this" then "look what happens"?
Improvising just for the sake of improvising seems to be presenting a self indulgent version of "look what I can do with my props." I suspect that most improvisation with card magic is some variation on find a card, lose a card, find wrong card and a color change.
If your magic is routined and rehearsed, you won't have any problems thinking what to perform next. Practice will help prevent making mistakes that will require an out. Also, most outs should be known (and specific to an effect) and practiced.
As a final thought, your audience wouldn't know if you are improvising. The chances your improvised performances are are not as good as something that has been practiced. Although you may think that improvising shows you are a good magician, your audience may think otherwise.