I once saw someone at the market alternating between two cards at the checkout. One card for groceries, the other for everything else. At first, it seemed excessive, but after calculating, it made sense.

Cards with annual fees initially seem like a loss. Why pay money that you don't have to?

However, cards that charge an annual fee are actually in the minority. According to WalletHub, out of over 1,500 card products, only about 15 percent have an annual fee.

On average, the annual fee for all cards is around $29.67. Most cards have no fee, which brings the average down.

This means that annual fee cards are not the default; they are designed for people with specific spending patterns.

The simplest calculation is for groceries. The Amex Blue Cash Preferred has a $95 annual fee but offers 6 percent back on up to $6,000 spent at U.S. supermarkets.

$6,000 translates to $500 a month. If you hit that limit, you get back $360, leaving you with $265 after the annual fee.

Of course, any amount over that limit drops to 1 percent. There are no unlimited great cards out there.

The comparison is with a no-annual-fee card that offers 3 percent back on groceries. The difference in rewards is 3 percentage points, so you need to spend about $3,167 annually to surpass the $95 annual fee.

That's about $264 a month. If your household spends more than $264 a month on groceries, the annual fee card is the better option.

Travel cards have a similar structure. The Chase Sapphire Preferred has a $95 annual fee but offers a $100 hotel credit for bookings made through Chase Travel on your card anniversary.

If you book a hotel through that channel just once, you've already recouped the annual fee. Plus, you earn 3 times points on dining and 5 times on Chase Travel bookings.

However, you must book the hotel through that specific channel. If you don't follow the rules, the $100 just disappears.

This is where the real crossroads begins. As the annual fee increases, the benefits often come in the form of credits that are broken down into smaller amounts.

The Amex Gold has a $325 annual fee and offers $120 in Uber Cash, $120 in dining credits, $100 in Resy credits, and $84 at Dunkin', totaling $424 in value.

On the surface, it looks like a good deal. The issue is that it requires you to spend a set amount at specific places each month.

The Chase Sapphire Reserve takes it a step further with a $795 annual fee. After subtracting a $300 travel credit, the effective cost is $495, and the rest involves concert tickets and designated hotel booking sites that you have to track down to use.

Every time I see these cards, I feel like I'm buying a coupon book. It's beneficial for those who diligently cut and use coupons, but for those who just stash them away, it's just a membership fee.

This isn't an unfounded concern. A Bankrate survey found that 23 percent of rewards card users reported not redeeming any rewards in the past year.

The reasons for not redeeming vary. About half said they were saving up, 11 percent didn't know how to use them, and 9 percent found the programs too complicated.

The more complex the benefits, the more they tend to favor those who have the time to manage that complexity. Card issuers are aware of this when designing their products.

In summary, the people who benefit from annual fee cards fall into one of three categories: those with clear spending in one category, those who have planned to use credits, or those who pay their balance in full each month.

Conversely, if you're carrying a balance into the next month, calculating the annual fee becomes meaningless. The interest will eat up the rewards entirely.

If I were in your shoes, I would start with a card that waives the annual fee for the first year. A product like the Blue Cash Preferred, which has a $0 annual fee in the first year, allows you to validate your actual spending over a year.

After a year, you can open your statement and do the math with a calculator. If the numbers don't exceed the annual fee, you can easily move on without regret.