Doesn't matter what his reason is unless you state specifically in your return policy (if you have one). While the guy may be an asshat, as a businessman your return policy is your word and reputation.
Would Target or Walmart question your Crabtree or any 49er jersey return-attempt the day after the Super Bowl?
As long as he's within his right via your return policy suck it up and take the card back.
Sometimes these returns are the cost of doing business. Any idea how many BIG televisions get returned to Best Buy the day after the Super Bowl?