Personally I think that you should wear whatever you like and whatever you're comfortable wearing. If you like it, don't worry about what others think.
There is one exception however. For the purposes of magic, you're goal is to amaze people. Now, depending on your image, and the type of person...