mlsanityData to 7 Oct 2026

DocMath-Eval: Evaluating Math Reasoning Capabilities of LLMs in Understanding Long and Specialized Documents

arXiv 2311.09805 ↗First posted 16 Nov 2023cs.CLcited by 45

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

Recent LLMs have demonstrated remarkable performance in solving exam-like math word problems. However, the degree to which these numerical reasoning skills are effective in real-world scenarios, particularly in expert domains, is still largely unexplored. This paper introduces DocMath-Eval, a comprehensive benchmark specifically designed to evaluate the numerical reasoning capabilities of LLMs in the context of understanding and analyzing specialized documents containing both text and tables. We conduct an extensive evaluation of 48 LLMs with Chain-of-Thought and Program-of-Thought prompting methods, aiming to comprehensively assess the capabilities and limitations of existing LLMs in DocMath-Eval. We found that even the current best-performing system (i.e., GPT-4o) still significantly lags behind human experts in solving complex numerical reasoning problems grounded in long contexts. We believe that DocMath-Eval can serve as a valuable benchmark for evaluating LLMs' capabilities in solving challenging numerical reasoning problems within expert domains.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 23 of its 53 references that are in the map. First posted on arXiv 16 Nov 2023.

Builds on

23 of 53 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning29 Sep

Other references · 22

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  5. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  6. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  7. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  8. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  9. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  10. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  11. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  12. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
10 more
  1. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  2. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 552
  3. 2021TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Financecited by 402
  4. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 358
  5. 2021A Diverse Corpus for Evaluating and Developing English Math Word Problem Solverscited by 314
  6. 2023TheoremQA: A Theorem-driven Question Answering datasetcited by 189
  7. 2022MultiHiertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual Datacited by 120
  8. 2022Large Language Models are few(1)-shot Table Reasonerscited by 116
  9. 2023FinGPT: Large Generative Models for a Small Languagecited by 49
  10. 2023Can GPT models be Financial Analysts? An Evaluation of ChatGPT and GPT-4 on mock CFA Examscited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 20; 2026: 16.

45 papers.

Sort:

By area:

  1. 2024FinBen: A Holistic Financial Benchmark for Large Language Modelscited by 140
  2. 2025PhysReason: A Comprehensive Benchmark towards Physics-Based Reasoningcited by 58
  3. 2024A Bounding Box is Worth One Token: Interleaving Layout and Text in a Large Language Model for Document Understandingcited by 57
  4. 2024Open-FinLLMs: Open Multimodal Large Language Models for Financial Applicationscited by 36
  5. 2024M3SciQA: A Multi-Modal Multi-Document Scientific QA Benchmark for Evaluating Foundation Modelscited by 26
  6. 2025Advancing Sequential Numerical Prediction in Autoregressive Modelscited by 26
  7. 2025MultiFinBen: Benchmarking Large Language Models for Multilingual and Multimodal Financial Applicationcited by 14
  8. 2025FinAuditing: A Financial Taxonomy-Structured Multi-Document Benchmark for Evaluating LLMscited by 14
  9. 2024MathHay: An Automated Benchmark for Long-Context Mathematical Reasoning in LLMscited by 13
  10. 2024Language Models are Symbolic Learners in Arithmeticcited by 11
  11. 2024SCITAT: A Question Answering Benchmark for Scientific Tables and Text Covering Diverse Reasoning Typescited by 10
  12. 2024Bridging Large Language Models and Optimization: A Unified Framework for Text-attributed Combinatorial Optimizationcited by 8
33 more
  1. 2025FinTrust: A Comprehensive Benchmark of Trustworthiness Evaluation in Finance Domaincited by 8
  2. 2024When Reasoning Meets Information Aggregation: A Case Study with Sports Narrativescited by 7
  3. 2026Who Deserves the Reward? SHARP: Shapley Credit-based Optimization for Multi-Agent Systemcited by 7
  4. 2025Knowledge Augmented Complex Problem Solving with Large Language Models: A Surveycited by 6
  5. 2025FinMMR: Make Financial Numerical Reasoning More Multimodal, Comprehensive, and Challengingcited by 6
  6. 2025Hierarchical Retrieval with Evidence Curation for Open-Domain Financial Question Answering on Standardized Documentscited by 4
  7. 2026GoLongRL: Capability-Oriented Long Context Reinforcement Learning with Multitask Alignmentcited by 4
  8. 2025Improving Table Understanding with LLMs and Entity-Oriented Searchcited by 3
  9. 2026VERA: Identifying and Leveraging Visual Evidence Retrieval Heads in Long-Context Understandingcited by 3
  10. 2026Conv-FinRe: A Conversational and Longitudinal Benchmark for Utility-Grounded Financial Recommendationcited by 3
  11. 2025eC-Tab2Text: Aspect-Based Text Generation from e-Commerce Product Tablescited by 2
  12. 2025MULTITAT: Benchmarking Multilingual Table-and-Text Question Answeringcited by 2
  13. 2025Supervised Fine-Tuning LLMs to Behave as Pedagogical Agents in Programming Educationcited by 2
  14. 2025SUCEA: Reasoning-Intensive Retrieval for Adversarial Fact-checking through Claim Decomposition and Editingcited by 2
  15. 2025Planning for Success: Exploring LLM Long-term Planning Capabilities in Table Understandingcited by 2
  16. 2025Merging Continual Pretraining Models for Domain-Specialized LLMs: A Case Study in Financecited by 2
  17. 2026Retrieval-Infused Reasoning Sandbox: A Benchmark for Decoupling Retrieval and Reasoning Capabilitiescited by 2
  18. 2026Rubric-Guided Process Reward for Stepwise Model Routingcited by 2
  19. 2026Beyond Teacher Likelihood: Group-Calibrated On-Policy Distillation for Long-Context Reasoningcited by 2
  20. 2025SCALAR: Scientific Citation-based Live Assessment of Long-context Academic Reasoningcited by 1
  21. 2025Understanding Financial Reasoning in AI: A Multimodal Benchmark and Error Learning Approachcited by 1
  22. 2026HVM-GraphRAG: Holistic-View Multimodal Graph Retrieval-Augmented Generation on Complex Documentcited by 1
  23. 2025Towards Competent AI for Fundamental Analysis in Finance: A Benchmark Dataset and Evaluationcited by 0
  24. 2025FinLFQA: Evaluating Attributed Text Generation of LLMs in Financial Long-Form Question Answeringcited by 0
  25. 2025Synthetic Data-Driven Prompt Tuning for Financial QA over Tables and Documentscited by 0
  26. 2026TableVista: Benchmarking Multimodal Table Reasoning under Visual and Structural Complexitycited by 0
  27. 2026How Do Document Parsers Break? Auditing Structural Vulnerability in Document Intelligencecited by 0
  28. 2026FinVerBench: Benchmark Validity and Calibration in Large Language Model Financial Statement Verificationcited by 0
  29. 2026MoCA-Agent: A Market-of-Claims Code Agent for Financial and Numerical Reasoningcited by 0
  30. 2026RusFinChain: A Russian Benchmark for Verifiable Chain-of-Thought Reasoning in Finance with Fuzzy-Aligned Evaluationcited by 0
  31. 2026KV-Pipe: On the Relation Between KV Sharing and Pipeline Parallel Efficiency in LLMscited by 0
  32. 2026When Tokenization is Secretly Output Supervisioncited by 0
  33. 2026DocMIDE: Learning Multi-Hop Implicit Derivation in Visually Rich Documentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Evaluating the Reasoning Abilities of LLMs on Underrepresented Mathematics Competition Problems
  2. 2025Do LLMs Overthink Basic Math Reasoning? Benchmarking the Accuracy-Efficiency Tradeoff in Language Models
  3. 2024Exposing the Achilles' Heel: Evaluating LLMs Ability to Handle Mistakes in Mathematical Reasoning
  4. 2024Evaluating LLMs' Mathematical Reasoning in Financial Document Question Answering
  5. 2025Can LLMs textitunderstand Math? -- Exploring the Pitfalls in Mathematical Reasoning
  6. 2025Brains vs. Bytes: Evaluating LLM Proficiency in Olympiad Mathematics
  7. 2026SenseMath: Do LLMs Have Number Sense? Evaluating Shortcut Use, Judgment, and Generation
  8. 2026The Effect of Scripts and Formats on LLM Numeracy