I agree with Christopher.I think the major issue that many people don't realize that improv is a skill that needs to be trained. They think when they get up in front of a crowd they'll just say whatever's on their mind. This is why so many of those performances are full of nothing, because suddenly the mind goes blank.
There's a huge difference between straight up improv, and being flexible in your presentation. Having a big ol' reservoir of one liners and such that you've honed over time and have developed the skill in using them, that's being flexible.
I most certainly do script my walk around and strolling performances. I know what I'm going to say for every trick. That doesn't mean I can't deviate from that script. But every time I do a certain trick, I'm going to say more or less the same thing. That kind of consistency is important for honing the effect of the trick. If you change more than one thing at a time, you don't know what's making your performance work or fail.
Although theater and magic share a lot in common, one major difference is that in magic you are directly interacting with your spectators. While scripting is extremely important and a useful tool, you need to be able to have a natural reaction to what your spectators are saying and respond accordingly. For example, if you are going to have a light conversation with friends you don't script every word beforehand.