# Credit Cards, Confusion, Computation, and Consequences: What Can We Uncover About Language Model Reasoning?

arXiv: https://arxiv.org/abs/2607.26952 · HTML: https://mlsanity.com/paper/2607.26952/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 19 of its 88 references that are in the map.
- First posted on arXiv 29 Jul 2026.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 19 of 19)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023)
- [Kimi K2: Open Agentic Intelligence](https://mlsanity.com/paper/2507.20534/) (2025)
- [GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models](https://mlsanity.com/paper/2508.06471/) (2025)
- [Alignment faking in large language models](https://mlsanity.com/paper/2412.14093/) (2024)
- [Large Language Models Often Know When They Are Being Evaluated](https://mlsanity.com/paper/2505.23836/) (2025)
- [DetermLR: Augmenting LLM-based Logical Reasoning from Indeterminacy to Determinacy](https://mlsanity.com/paper/2310.18659/) (2023)
- [Ax-Prover: A Deep Reasoning Agentic Framework for Theorem Proving in Mathematics and Quantum Physics](https://mlsanity.com/paper/2510.12787/) (2025)
- [Zero is Not Hero Yet: Benchmarking Zero-Shot Performance of LLMs for Financial Tasks](https://mlsanity.com/paper/2305.16633/) (2023)
- [MultiFinBen: Benchmarking Large Language Models for Multilingual and Multimodal Financial Application](https://mlsanity.com/paper/2506.14028/) (2025)
- [Benchmarking LLMs' Mathematical Reasoning with Unseen Random Variables Questions](https://mlsanity.com/paper/2501.11790/) (2025)
- [Population-Evolve: a Parallel Sampling and Evolutionary Method for LLM Math Reasoning](https://mlsanity.com/paper/2512.19081/) (2025)
- [NUMINA: A Natural Understanding Benchmark for Multi-dimensional Intelligence and Numerical Reasoning Abilities](https://mlsanity.com/paper/2509.16656/) (2025)
- [Adaptive Selection of Symbolic Languages for Improving LLM Logical Reasoning](https://mlsanity.com/paper/2510.10703/) (2025)
- [Multi-LLM Collaboration for Medication Recommendation](https://mlsanity.com/paper/2512.05066/) (2025)
- [Rectify Evaluation Preference: Improving LLMs' Critique on Math Reasoning via Perplexity-aware Reinforcement Learning](https://mlsanity.com/paper/2511.10303/) (2025)

## Abstract (arXiv)

We introduce CreditCardQA, the first financial literacy benchmark for numerical reasoning derived from real credit card agreements. The dataset contains 1,800 questions, including first-person variants that reflect how consumers naturally ask about fees, interest, and payments. We evaluate a range of large language and reasoning models under Chain-of-Thought (CoT) and Program-of-Thought (PoT) prompting. Overall, PoT yields consistent performance gains, particularly for models with weaker baseline reasoning, and narrows gaps between open- and closed-source systems. Through error analysis, we show that failures arise less from arithmetic and more from misapplied financial rules, missed conditions, and misunderstandings of contractual terms. We further analyze question difficulty and find that comparisons, conditional logic, and monetary constraints are especially challenging. We also find that errors often arise in edge cases such as late-payment penalties or small-balance scenarios that are more likely to affect lower-income or financially vulnerable individuals.