After checking with the online magic shops that carry the effect to no avail...I decided to, one more time, watch the effect. I noticed something I hadn't noticed before. During one of the counts, I realized "Hey fool...that count would not be possible without one more card." So I added the 6th card....voila! I remembered it!
What was happening is that I was remembering it all along. I was always doing the two different counts properly. The problem was that because I didn't have the 6th card, it was not working anymore. So I assumed I was forgetting a move. I was in fact forgetting "something" but a move wasn't it lol.