Here's some advice, as I've been both a performing magician and nightclub doorman/bouncer.
Fighting is never the answer, and if he swore at you, and you hit him, then you're the aggressor. It's the actions of a bully. It's also GBH with intent, which can carry a prison sentence for repeat...