A magician can do tricks. A showman tips the scales between tricks and witnessing something within an inch of true magic. Guess which one of those has put the work in to properly dialoged patter.
So yes, I think patter is really important...if you do magic for money. As a hobbyist I wouldn't...