It's the whole "sell yourself before the trick". You should really work on connecting to them on a person-to-person level, not as magician-and-spectator. Pretend you don't have any magic, you don't even do magic, then go up to these people and start a conversation. Be confident and dominant, but not irritating.
One of my good friends, myself and some others were hanging out at a hobby store when he came in. This guy's got a strong charismatic air about him - he's playful, flirtatious, bold, and quirky. He owns the room. He doesn't care if you like him or not (though he aims for like), he just wants everyone to know he's arrived. He's also a really great guy. He came up to us and just sat down and got into a conversation. We all got that "stranger-anxiety" at first, but when we saw the guy just wanted to talk and had interesting things to say, and invited us to come back a different day to hang out, it worked out. He turned out to be one of my very best friends.
My general philosophy is this - approach people, as said above, as if you weren't going to perform anything. Be like-able, interesting, and presentable enough to sustain a conversation without magic. Hunt for people you'd like to talk to, and who you think you could get along with. Start with like-minded folk, then move on to more challenging game: people who appear to be total opposites. Actually go out and meet new people, have fun with it. Be confident, be in control, and present your audience with a really cool and intriguing person. Someone colorful and fun.
And, yeah, you should be able to sustain conversation about non-magical things. And, yeah, it needs to last a few.
Once they're enamored with you, then bring out the magic.