I know its not audience management because I tell him exactly what to do and everything.
Actually, part of it could be audience management. Think about what would make him want to count the cards. Was there something in the initial count of the cards that made him suspicious?
As
@MohanaMisra said, even executing a slight well may raise suspicion if your body language communicates that something is happening.
Was there something in your presentation that tipped him off that the number of cards he was holding might change? In this effect, you want them to count the cards but at the right time. What did you say when you asked him to hold the cards? If you said "hold the cards and don't do anything else with them" that is suspicious. If you said "hold the cards where everyone can see them because from this point on, I don't want to touch them", that is better.
Having someone hold cards against their chest is awkward... it makes them want to look at them. Have them put a palm out, you put the cards on their palm and then have them put their other hand on top. Adding, "I want you to stay just like that" helps. That position works better because their reaction is to stare at their hands. Try holding cards both ways and see if you feel differently.
Avoid saying any negatives - words like "don't move" "don't look" "don't count" just make the person want to do what you told them not to do. Yes, we are all toddlers at heart when we are told "don't."
Also, you have to make the number of cards seem random. If you emphasize the number ten too much, then a spectator will become suspicious and want to know if there really is ten.
Audience management is more than just telling them what to do. It is guiding their thoughts and emotions in a way that they cooperate with you. Often difficult spectators give us a gift in that they help us realize what we could do better.
I know its not because of a competitive attitude.
Well, it may be and you don't realize it. Think about this question -- is the goal of your performance of magic for spectator's to see amazing things that you can do? I'm guessing most magicians answer yes. The problem is that the goal of your performance should be to provide your audience with a magical experience.
Part of the problem is that most magicians use say-do-see presentations. They say what they are going to do, do it and then tell the audience to "see" what the result is. If you just talk about "what" you are doing, the audience is going to think about "how" you are doing it. If the focus is on you and what you are doing, the audience will perceive that as being competitive.
I think I should just stop preforming for him.
If he is a willing spectator, it might be worth the effort to keep performing because if you develop pieces that you can perform and get a good reaction from him, you could get a good reaction from anyone. Think of it as a challenge.