Are the Financial Reasoning from LLMs Credible? A Real World Test over Long-Horizon Statements
On the map: LLM reasoning → Reasoning for agents, planning and code
Abstract (arXiv)
Do Large Language Models (LLMs) possess genuine structural reasoning, or merely rely on surface-level pattern matching? The financial domain, demanding numerical precision and multi-step logic over long contexts, is an ideal testbed. Existing benchmarks fail to capture real-world industrial complexity, predominantly relying on multiple-choice questions or single-hop QA over cropped tables while ignoring intricate cross-statement dynamics and temporal de-cumulation. To bridge this gap, we introduce FinIndices, a large-scale benchmark evaluating data-processing fidelity over uncropped financial statements (up to 32K tokens). Utilizing an automated synthesis pipeline with adversarial traps, FinIndices encompasses Single-Index computation and Table-Index tabulation to test complex domain, temporal, and caliber reasoning. Our evaluation reveals two severe LLM vulnerabilities. First, a "Knowledge Bottleneck": despite memorizing formulas during pre-training, models demonstrate fragile pattern matching. Removing explicit formula hints causes performance to collapse (e.g., Gemini-3.1-Pro drops from 70.70% to 38.22% on table tasks), exposing fatal flaws in temporal de-cumulation and stock-flow caliber mismatch. Second, a "Structural Bottleneck": the intense cognitive load of generating multi-metric, multi-period tables actively drains reasoning capacity. Under structural pressure, LLMs that flawlessly execute isolated derivations regress to shallow heuristics, such as fetching incorrect adjacent columns or substituting deep accounting adjustments with lazy literal arithmetic. Finally, Supervised Fine-Tuning (SFT) yields substantial zero-hint gains (+8.54% Single, +3.82% Table), validating that structured logic can be partially restored via data-centric alignment.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 15 of its 17 references that are in the map. First posted on arXiv 22 Jul 2026.
Builds on
15 of 17 references.
Closest ancestors · 7
Same direction, newest first
- 2026FIRE: A Comprehensive Benchmark for Financial Intelligence and Reasoning Evaluation25 Feb
- 2025Agentar-Fin-R1: Enhancing Financial Intelligence through Domain Expertise, Training Efficiency, and Advanced Reasoning22 Jul
- 2025FinEval-KR: A Financial Domain Evaluation Framework for Large Language Models' Knowledge and Reasoning18 Jun
- 2025FinanceReasoning: Benchmarking Financial Numerical Reasoning More Credible, Comprehensive and Challenging6 Jun
- 2025FinMME: Benchmark Dataset for Financial Multi-Modal Reasoning Evaluation30 May
- 2025Fin-R1: A Large Language Model for Financial Reasoning through Reinforcement Learning20 Mar
- 2023FinanceMath: Knowledge-Intensive Math Reasoning in Finance Domains16 Nov
Other references · 8
- 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
- 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 552
- 2021TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Financecited by 402
- 2023FinanceBench: A New Benchmark for Financial Question Answeringcited by 261
- 2025Finance Agent Benchmark: Benchmarking LLMs on Real-world Financial Research Taskscited by 63
- 2025BizFinBench: A Business-Driven Real-World Financial Benchmark for Evaluating LLMscited by 22
- 2025FLAME: Financial Large-Language Model Assessment and Metrics Evaluationcited by 8
- 2025Expect the Unexpected: FailSafe Long Context QA for Financecited by 7
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026V-FiLLM: Verified Financial LLM Reasoning Benchmark
- 2023Beyond Classification: Financial Reasoning in State-of-the-Art Language Models
- 2026FinRule-Bench: A Benchmark for Joint Reasoning over Financial Tables and Principles
- 2025XFinBench: Benchmarking LLMs in Complex Financial Problem Solving and Reasoning
- 2025Evaluating Large Language Models for Financial Reasoning: A CFA-Based Benchmark Study
- 2026Are Large Language Models Reliable Reviewers? A Benchmark for Error Detection in Financial Documents
- 2026FinMCP-Bench: Benchmarking LLM Agents for Real-World Financial Tool Use under the Model Context Protocol
- 2026FinSheet-Bench: From Simple Lookups to Complex Reasoning, Where LLMs Break on Financial Spreadsheets