I think impromptu effects are great, but having things on you at all times is a must at some point. As I become better known in my area I realized that I need to be "on" all the time.
Magician's aren't like other art forms, a sketch artist would be the closest comparison maybe. We have to be what we claim to be all the time. Dancers don't always have their apparel and music, musicians - their instruments, etc. People don't separate magicians like they do other artists from their art form. They want to see something here and now, and why shouldn't they?
"I don't have anything on me" is a lame excuse.
Having your business cards (out to lunch included) is something I would consider impromptu, same with card effects, coin effects... anything you can carry on you, you probably should. Rubber bands, lighters and anything else you know effects with.
Impromptu performances are pretty unrealistic. If you claim yourself a magician, there's hardly an excuse to not be prepared to do anything.