# Are the Financial Reasoning from LLMs Credible? A Real World Test over Long-Horizon Statements

arXiv: https://arxiv.org/abs/2607.28661 · HTML: https://mlsanity.com/paper/2607.28661/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 15 of its 17 references that are in the map.
- First posted on arXiv 22 Jul 2026.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 15 of 15)

- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [FinQA: A Dataset of Numerical Reasoning over Financial Data](https://mlsanity.com/paper/2109.00122/) (2021)
- [TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance](https://mlsanity.com/paper/2105.07624/) (2021)
- [FinanceBench: A New Benchmark for Financial Question Answering](https://mlsanity.com/paper/2311.11944/) (2023)
- [Fin-R1: A Large Language Model for Financial Reasoning through Reinforcement Learning](https://mlsanity.com/paper/2503.16252/) (2025)
- [Finance Agent Benchmark: Benchmarking LLMs on Real-world Financial Research Tasks](https://mlsanity.com/paper/2508.00828/) (2025)
- [FinanceMath: Knowledge-Intensive Math Reasoning in Finance Domains](https://mlsanity.com/paper/2311.09797/) (2023)
- [FinMME: Benchmark Dataset for Financial Multi-Modal Reasoning Evaluation](https://mlsanity.com/paper/2505.24714/) (2025)
- [FinanceReasoning: Benchmarking Financial Numerical Reasoning More Credible, Comprehensive and Challenging](https://mlsanity.com/paper/2506.05828/) (2025)
- [BizFinBench: A Business-Driven Real-World Financial Benchmark for Evaluating LLMs](https://mlsanity.com/paper/2505.19457/) (2025)
- [FLAME: Financial Large-Language Model Assessment and Metrics Evaluation](https://mlsanity.com/paper/2501.06211/) (2025)
- [Expect the Unexpected: FailSafe Long Context QA for Finance](https://mlsanity.com/paper/2502.06329/) (2025)
- [Agentar-Fin-R1: Enhancing Financial Intelligence through Domain Expertise, Training Efficiency, and Advanced Reasoning](https://mlsanity.com/paper/2507.16802/) (2025)
- [FIRE: A Comprehensive Benchmark for Financial Intelligence and Reasoning Evaluation](https://mlsanity.com/paper/2602.22273/) (2026)
- [FinEval-KR: A Financial Domain Evaluation Framework for Large Language Models' Knowledge and Reasoning](https://mlsanity.com/paper/2506.21591/) (2025)

## Abstract (arXiv)

Do Large Language Models (LLMs) possess genuine structural reasoning, or merely rely on surface-level pattern matching? The financial domain, demanding numerical precision and multi-step logic over long contexts, is an ideal testbed. Existing benchmarks fail to capture real-world industrial complexity, predominantly relying on multiple-choice questions or single-hop QA over cropped tables while ignoring intricate cross-statement dynamics and temporal de-cumulation. To bridge this gap, we introduce FinIndices, a large-scale benchmark evaluating data-processing fidelity over uncropped financial statements (up to 32K tokens). Utilizing an automated synthesis pipeline with adversarial traps, FinIndices encompasses Single-Index computation and Table-Index tabulation to test complex domain, temporal, and caliber reasoning. Our evaluation reveals two severe LLM vulnerabilities. First, a "Knowledge Bottleneck": despite memorizing formulas during pre-training, models demonstrate fragile pattern matching. Removing explicit formula hints causes performance to collapse (e.g., Gemini-3.1-Pro drops from 70.70% to 38.22% on table tasks), exposing fatal flaws in temporal de-cumulation and stock-flow caliber mismatch. Second, a "Structural Bottleneck": the intense cognitive load of generating multi-metric, multi-period tables actively drains reasoning capacity. Under structural pressure, LLMs that flawlessly execute isolated derivations regress to shallow heuristics, such as fetching incorrect adjacent columns or substituting deep accounting adjustments with lazy literal arithmetic. Finally, Supervised Fine-Tuning (SFT) yields substantial zero-hint gains (+8.54% Single, +3.82% Table), validating that structured logic can be partially restored via data-centric alignment.