mlsanityData to 14 Oct 2026

Evaluating LLMs' Mathematical Reasoning in Financial Document Question Answering

arXiv 2402.11194 ↗First posted 17 Feb 2024cs.CLcited by 25

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Large Language Models (LLMs), excel in natural language understanding, but their capability for complex mathematical reasoning with an amalgamation of structured tables and unstructured text is uncertain. This study explores LLMs' mathematical reasoning on four financial tabular question-answering datasets: TATQA, FinQA, ConvFinQA, and Multihiertt. Through extensive experiments with various models and prompting techniques, we assess how LLMs adapt to complex tables and mathematical tasks. We focus on sensitivity to table complexity and performance variations with an increasing number of arithmetic reasoning steps. The results provide insights into LLMs' capabilities and limitations in handling complex mathematical scenarios for semi-structured tables. Ultimately, we introduce a novel prompting technique tailored to semi-structured documents, matching or outperforming other baselines in performance while providing a nuanced understanding of LLMs abilities for such a task.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 34 of its 39 references that are in the map. First posted on arXiv 17 Feb 2024.

Builds on

34 of 39 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering7 Oct

Other references · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  5. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  6. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  7. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  8. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
  9. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
  10. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 662
  11. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  12. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 562
21 more
  1. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 558
  2. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 487
  3. 2020TAPAS: Weakly Supervised Table Parsing via Pre-trainingcited by 434
  4. 2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 428
  5. 2021TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Financecited by 408
  6. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 388
  7. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 350
  8. 2020HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Datacited by 264
  9. 2021TAPEX: Table Pre-training via Learning a Neural SQL Executorcited by 235
  10. 2021MultiModalQA: Complex Question Answering over Text, Tables and Imagescited by 179
  11. 2023Large Language Models are Versatile Decomposers: Decompose Evidence and Questions for Table-based Reasoningcited by 178
  12. 2020Open Question Answering over Tables and Textcited by 163
  13. 2022MultiHiertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual Datacited by 121
  14. 2022Large Language Models are few(1)-shot Table Reasonerscited by 116
  15. 2022Successive Prompting for Decomposing Complex Questionscited by 94
  16. 2020INFOTABS: Inference on Tables as Semi-structured Datacited by 86
  17. 2022NaturalProver: Grounded Mathematical Proof Generation with Language Modelscited by 72
  18. 2022Learning Math Reasoning from Self-Sampled Correct and Partially-Correct Solutionscited by 38
  19. 2019From 'F' to 'A' on the N.Y. Regents Science Exams: An Overview of the Aristo Projectcited by 33
  20. 2021MWP-BERT: Numeracy-Augmented Pre-training for Math Word Problem Solvingcited by 32
  21. 2016Annotating Derivations: A New Evaluation Strategy and Dataset for Algebra Word Problemscited by 28

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 11; 2026: 7.

25 papers.

Sort:

By area:

  1. 2024A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challengescited by 55
  2. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  3. 2025PR-Attack: Coordinated Prompt-RAG Attacks on Retrieval-Augmented Generation in Large Language Models via Bilevel Optimizationcited by 32
  4. 2024AI in Investment Analysis: LLMs for Equity Stock Ratingscited by 10
  5. 2025Rethinking Table Instruction Tuningcited by 10
  6. 2024Enhancing Temporal Understanding in LLMs for Semi-structured Tablescited by 8
  7. 2024AGENTiGraph: An Interactive Knowledge Graph Platform for LLM-based Chatbots Utilizing Private Datacited by 8
  8. 2025Plugging Schema Graph into Multi-Table QA: A Human-Guided Framework for Reducing LLM Reliancecited by 7
  9. 2024DeFine: Decision-Making with Analogical Reasoning over Factor Profilescited by 4
  10. 2026Unlocking Data Value in Finance: A Study on Distillation and Difficulty-Aware Trainingcited by 4
  11. 2025Extract, Match, and Score: An Evaluation Paradigm for Long Question-context-answer Triplets in Financial Analysiscited by 3
  12. 2025Evaluating Large Language Models for Financial Reasoning: A CFA-Based Benchmark Studycited by 3
13 more
  1. 2026FinSafetyBench: Evaluating LLM Safety in Real-World Financial Scenarioscited by 3
  2. 2025Merging Continual Pretraining Models for Domain-Specialized LLMs: A Case Study in Financecited by 2
  3. 2024MATATA: Weakly Supervised End-to-End MAthematical Tool-Augmented Reasoning for Tabular Applicationscited by 1
  4. 2025What Factors Affect LLMs and RLLMs in Financial Question Answering?cited by 1
  5. 2025Fin-Ally: Pioneering the Development of an Advanced, Commonsense-Embedded Conversational AI for Money Matterscited by 1
  6. 2024Learning to Solve Domain-Specific Calculation Problems with Knowledge-Intensive Programs Generatorcited by 0
  7. 2025Interpreting Multi-Attribute Confounding through Numerical Attributes in Large Language Modelscited by 0
  8. 2025Accounting Reasoning in Large Language Models: Concepts, Evaluation, and Empirical Analysiscited by 0
  9. 2026Evaluating Accounting Reasoning Capabilities of Large Language Modelscited by 0
  10. 2026QUIETT: Query-Independent Table Transformation for Robust Reasoningcited by 0
  11. 2026ASTRA: Adaptive Semantic Tree Reasoning Architecture for Complex Table Question Answeringcited by 0
  12. 2026From Documents to Reasoning: A Validated Synthetic Data Pipeline and Semantic-Aware Fine-Tuning for Financial Numerical Reasoningcited by 0
  13. 2026When Tokenization is Secretly Output Supervisioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MathHay: An Automated Benchmark for Long-Context Mathematical Reasoning in LLMs
  2. 2026Structure First, Reason Next: Enhancing a Large Language Model using Knowledge Graph for Numerical Reasoning in Financial Documents
  3. 2025Large Language Models and Mathematical Reasoning Failures
  4. 2023Beyond Classification: Financial Reasoning in State-of-the-Art Language Models
  5. 2023FinanceMath: Knowledge-Intensive Math Reasoning in Finance Domains
  6. 2025FinEval-KR: A Financial Domain Evaluation Framework for Large Language Models' Knowledge and Reasoning
  7. 2026Are the Financial Reasoning from LLMs Credible? A Real World Test over Long-Horizon Statements
  8. 2024LLM Reasoning Engine: Specialized Training for Enhanced Mathematical Reasoning