i agree, morgician nailed it.
the real awe of magic and especially sleight of hand magic, is to make the illusions look simple and straight forward. you want people to think that NOTHING has happened when you are already steps ahead.
The problem with flourishing during magic is that the audience will attribute what they have seen to skill and speed. no matter how impossible the effect or invisible the sleight, if they know you have skills with card manipulation, they will turn to "fast hands" as the only answer and that ruins the illusion and mystery.
as morgician said, try it for yourself. learn an effect that is straight forward and elegent, with no fancy moves and only hidden and indetectable sleights, and see the reactions you get when you pull the card out of an impossible location.
then do the same effect for a different group, but start with some flourishes first and see what kind of reactions people give you. Like morgician said, they will be technically impressed with your skills, but you wont leave a magical impression.
I do agree with you to a certain extent, but there are many ways where you can implement flourishes into magic. Ricky Jay does that very well, especially the waterfall card revelation. I recommending doing flourishy reveals when performing to people of high authorities, as Lee Asher points out in Flo Cascade reveal.