(and I'm doing effects like prophet and invisible deck so I know it's not my material).
No, it's not the material. It's you.
The material you perform is largely irrelevant - As long as it's appropriate to the venue/audience. All that really matters beyond that is your performance of that material. Which, unfortunately, means that the problem you're encountering is that you are not engaging and entertaining these audiences.
The first thing you need to learn is how to engage someone in conversation without magic as a crutch. The easiest way to do this is developing the habit of chatting with strangers. When you're waiting in line, pick out something notable about someone near you and comment on that. Then introduce yourself, and ask a question. Compliment, introduction, question. Example: "Hey, that's a really nice necklace." "Thanks!" "My name is Luke. Where did you find that necklace?" ... profit! (That's a really summarized version of that conversation, btw).
Once you get comfortable with engaging strangers in conversation all you need to do is that - Engage strangers in conversation, then steer the conversation to a subject that will let you naturally segue into performance.