I am surprised no one has mentioned the fact that adults in general would rather talk to another adult than a 13 year old, and card magic is by far one of the least entertaining, stereotypical types of magic (don't get me wrong cards are my favorite). That is one of the big reasons, in my opinion, why you don't want to open any performance with cards in your hand.
So with the two things I mentioned (age and cards) you have 2 things going against you.
Casen, you mentioned that you heard what they were talking about as they walked away and it wasn't important, to you. I think another thing you are not seeing is the fact that things talked about by adults will not be important to a 13 year old. Think about it this way, if you are showing an adult a trick and another adult walks up and says, "You need to check out what I found behind the church". It isn't interesting or important to you, but there might have been graffiti or something that could put the people in the church in danger if they went outside the church, or something to that extent.
In this instance, the only one rude in what happened was the person that came up and interrupted your performance. That is where audience control comes into play. If it were me I would have handled it the same way Chris Mayhew said to handle it. Make them part of the effect so they know your pastor was already preoccupied watching something else.
As I typed that last paragraph I now have a couple more thoughts. Personally, I think the guy that "interrupted" may not have been rude in "interrupting" your performance. This is why, do you think the guy "interrupted" your performance because he didn't even know that you were performing when he came up and started talking to the pastor? People watching magic normally don't talk very much until the performance is over. If I walked up behind someone that I needed to talk to and saw that they were not talking to anyone else I would start talking to them. Before you can even say the guy saw you standing there performing, the guy that interrupted may NOT have noticed you were standing there performing for your pastor (adults don't always pay attention to kids the way you think they do or should).
I am not running you down by saying what I said, but you need to look at things from the adult laymen point of view not a magician or child point of view.