Most of the time, they will look where you look, or direct them to look. I went to an Aaron Fisher Lecture and when he started explaining how an effect was done, he pointed out the moments when he basically said "hey, look waaaay over there!" and we all obeyed. When you have someone select a card and tell everyone to remember it, you're saying just that. You're telling them to look totally away from you, and then to memorize an object. That's the perfect time to do ANYTHING you want. Try and structure your effects to where when they do select a card, you are able to do a sleight to progress the trick. Think about this, you have a golf ball and do a fake transfer to make it vanish. If you follow your hand that is "holding" the ball and pay no attention to your other one (because it's empty, so why would it be important, right?) then your audience will do the same. Look where you want them to look, don't look where you don't want them to look. Also body language is very important. I know Greg Wilson talked about how Slydini would relax in his chair, shoulders untensed, relaxed breathing, no tension in his body. The audience then reacted the same way, because the trick was over. And in that moment he would do whatever secret move that needed to be done, because there is no heat on him, everyone and everything is relaxed. It's important to create on beats and off beats in your magic (easier said than done, I know) so you have that moment where the tension is released and you can do a sleight with no pressure on you.
I hope that all makes sense and you can take something away from that.
Jacob