I think you pose an interesting question, and you have an interesting experience.
From my own experience, a few things come to mind...
1) I pride myself on my friends. I love them dearly - I would consider myself to have a large number of friends, but my "inner circle(s)" so to speak are few. But the friends I do choose to have in that circle know me well - and I know them well - and I know that none of them would deliberately do something like that just to be a "bad guy" or for the hell of it - they're all extremely good people. So, answer one, it may just be the person (not to criticise or otherwise reflect badly on your friends or anything - that's not my place, but definitely be aware obviously that some people are more polite about these things than others).
2) I am keen to stress to everyone that I perform for how much it means to me. I'm serious about what I do (although note that this does not mean my performances are always serious!), and it shows. I do what I love - I perform mentalism because I have a genuine, real life, passion for psychological knowledge which would exist if my passion for mentalism did not, and indeed predates me picking up mentalism. This is important because I base my performances on real psychological phenomena, that really interest me, and that I love talking about. I could easily talk for an hour on various psychological phenomena that interest me, just as I could perform them. So, this passion and respect that I give what I do, I'd like to think, shows. It's exciting for me to perform effects and watch reactions - but it's also exciting for me to talk about these things in my patter during what would otherwise be dead time too! Psychology has played a very important part in my life (I know that's a very general statement, but it's changed who I am dramatically), and as such, I respect it. This sort of thing shows. If you perform tricks as tricks, then they will be treated as such - tricks. If you respect your magic, it will show as well. Obviously I don't know you or how you perform, but that's another potential answer.
3) Another thing mentioned above is how presentation dependent it will be. Obviously, as a mentalist, I could never pass for a psychic. I can't pull off that persona, because everyone knows that I can't read minds, I'm not a medium, I don't talk to the dead, and so forth. But, my friends also do know that I'm a psychology major. Some know of my history with abnormal psychology. They also know that I read widely on these things. So when I present it - it's realistic. I don't claim to have magical powers, or parapsychological powers, which they could call me out on. Rather, I perform realistic but nonetheless astonishing mentalism, that I don't get called out on.
As a side note, it's a little difficult and different for me too because I progressed from sleight of hand card magic to mentalism - therefore, they already know I can manipulate cards and so forth - so I have to be very careful in differentiating one from the other. I can show sleight of hand magic, and do, but when I do, it's very clearly differentiated.
Anyway - obviously, if you present sleight of hand as magical powers ("Watch - if I wave my hand, the card comes to the top" - it doesn't take a genius to know that you waving your hand did nothing), then by virtue of elimination, your performance has already degraded into a puzzle ("He has no magical powers. Therefore, he must be using sleight of hand. I KNOW he's using sleight of hand. I wonder how he does that?") Therefore, they will be much more likely to do something like this - and feel more comfortable doing it because they know you. Strangers will not do it as often, because we are more liberal with those we know than those we don't. To get around this, you have to approach magic in a different theoretical and presentational mindset - and there ARE a few ways to do this. Just depends what suits.