A bit of both. Sometimes we play with an idea, show it to a few people and decide if it's actually deceptive. Often it's our skill level, or performance and patter choices that make it deceptive. Sometimes, it's just a good or bad trick. At one point in my favorite card routine, there is a color change, and people spaz out. This one time I performed it for a family with a young girl (probably 4 years old). The adults started yelling, and the little girl just looked confused and said "It didn't change. You just turned it over". So funny.
Some spectators are just wise and can deduce a logical way it was done. A great read on this topic is covered in the book 'Strong Magic' by Darwin Ortiz. He talks about how spectators can figure out a small part of the workings of an effect, and then they FEEL like they have the entire thing figured out. For instance they might see you palm a card for a brief moment, and have no idea how you made the card appear inside a locked box, inside another locked box, inside a locked safe, inside an elephant. But they will claim they know because they saw 'how you did it'. The don't know, but they FEEL like they do. (you may have not even palmed a card, but if they THINK you did, they feel confident that they know the secret.
It should absolutely motivate you to practice and be more deceptive. It should also prompt you to explore 'Why' they know the secret to an effect. Friends can be great because, 1) you aren't trying to convince them that you are actually magic, you all can admit there is a method. and 2) they can be honest if they saw you flash or how they came to the conclusion on method. Sometimes our words and movement can help. My two favorite examples are the vanishing silk (using a TT) and the taking your thumb off trick. When I do the vanishing silk, I actually have people say "Oh I saw someone do that, but they had a fake finger". That indicates to me that, it was deceptive, they didn't see my method, and they think that whatever I did, it wasn't what they immediately had thought of. The thumb trick is a little different. People know how that works, but the way I do it, makes them take pause and do a double take and makes them question what they think they saw -if even for a short moment. I should put up a video of that.....