# DocMath-Eval: Evaluating Math Reasoning Capabilities of LLMs in Understanding Long and Specialized Documents

arXiv: https://arxiv.org/abs/2311.09805 · HTML: https://mlsanity.com/paper/2311.09805/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Placed in Mathematical reasoning, one of 17 directions in LLM reasoning.
- Builds on 23 of its 53 references that are in the map.
- First posted on arXiv 16 Nov 2023.

## Placed in

- LLM reasoning / [Mathematical reasoning](https://mlsanity.com/lr/mathematical-reasoning/)

## Builds on (most cited first, 23 of 23)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks](https://mlsanity.com/paper/2211.12588/) (2022)
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://mlsanity.com/paper/1903.00161/) (2019)
- [Are NLP Models really able to Solve Simple Math Word Problems?](https://mlsanity.com/paper/2103.07191/) (2021)
- [WizardCoder: Empowering Code Large Language Models with Evol-Instruct](https://mlsanity.com/paper/2306.08568/) (2023)
- [MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms](https://mlsanity.com/paper/1905.13319/) (2019)
- [Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problems](https://mlsanity.com/paper/1705.04146/) (2017)
- [FinQA: A Dataset of Numerical Reasoning over Financial Data](https://mlsanity.com/paper/2109.00122/) (2021)
- [TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance](https://mlsanity.com/paper/2105.07624/) (2021)
- [Text and Code Embeddings by Contrastive Pre-Training](https://mlsanity.com/paper/2201.10005/) (2022)
- [Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning](https://mlsanity.com/paper/2209.14610/) (2022)
- [A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers](https://mlsanity.com/paper/2106.15772/) (2021)
- [TheoremQA: A Theorem-driven Question Answering dataset](https://mlsanity.com/paper/2305.12524/) (2023)
- [MultiHiertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual Data](https://mlsanity.com/paper/2206.01347/) (2022)
- [Large Language Models are few(1)-shot Table Reasoners](https://mlsanity.com/paper/2210.06710/) (2022)
- [FinGPT: Large Generative Models for a Small Language](https://mlsanity.com/paper/2311.05640/) (2023)
- [Can GPT models be Financial Analysts? An Evaluation of ChatGPT and GPT-4 on mock CFA Exams](https://mlsanity.com/paper/2310.08678/) (2023)

## Built on by (most cited first, 30 of 45)

- [FinBen: A Holistic Financial Benchmark for Large Language Models](https://mlsanity.com/paper/2402.12659/) (2024)
- [PhysReason: A Comprehensive Benchmark towards Physics-Based Reasoning](https://mlsanity.com/paper/2502.12054/) (2025)
- [A Bounding Box is Worth One Token: Interleaving Layout and Text in a Large Language Model for Document Understanding](https://mlsanity.com/paper/2407.01976/) (2024)
- [Open-FinLLMs: Open Multimodal Large Language Models for Financial Applications](https://mlsanity.com/paper/2408.11878/) (2024)
- [M3SciQA: A Multi-Modal Multi-Document Scientific QA Benchmark for Evaluating Foundation Models](https://mlsanity.com/paper/2411.04075/) (2024)
- [Advancing Sequential Numerical Prediction in Autoregressive Models](https://mlsanity.com/paper/2505.13077/) (2025)
- [MultiFinBen: Benchmarking Large Language Models for Multilingual and Multimodal Financial Application](https://mlsanity.com/paper/2506.14028/) (2025)
- [FinAuditing: A Financial Taxonomy-Structured Multi-Document Benchmark for Evaluating LLMs](https://mlsanity.com/paper/2510.08886/) (2025)
- [MathHay: An Automated Benchmark for Long-Context Mathematical Reasoning in LLMs](https://mlsanity.com/paper/2410.04698/) (2024)
- [Language Models are Symbolic Learners in Arithmetic](https://mlsanity.com/paper/2410.15580/) (2024)
- [SCITAT: A Question Answering Benchmark for Scientific Tables and Text Covering Diverse Reasoning Types](https://mlsanity.com/paper/2412.11757/) (2024)
- [Bridging Large Language Models and Optimization: A Unified Framework for Text-attributed Combinatorial Optimization](https://mlsanity.com/paper/2408.12214/) (2024)
- [FinTrust: A Comprehensive Benchmark of Trustworthiness Evaluation in Finance Domain](https://mlsanity.com/paper/2510.15232/) (2025)
- [When Reasoning Meets Information Aggregation: A Case Study with Sports Narratives](https://mlsanity.com/paper/2406.12084/) (2024)
- [FinMMR: Make Financial Numerical Reasoning More Multimodal, Comprehensive, and Challenging](https://mlsanity.com/paper/2508.04625/) (2025)
- [Who Deserves the Reward? SHARP: Shapley Credit-based Optimization for Multi-Agent System](https://mlsanity.com/paper/2602.08335/) (2026)
- [Knowledge Augmented Complex Problem Solving with Large Language Models: A Survey](https://mlsanity.com/paper/2505.03418/) (2025)
- [GoLongRL: Capability-Oriented Long Context Reinforcement Learning with Multitask Alignment](https://mlsanity.com/paper/2605.19577/) (2026)
- [Hierarchical Retrieval with Evidence Curation for Open-Domain Financial Question Answering on Standardized Documents](https://mlsanity.com/paper/2505.20368/) (2025)
- [Supervised Fine-Tuning LLMs to Behave as Pedagogical Agents in Programming Education](https://mlsanity.com/paper/2502.20527/) (2025)
- [Conv-FinRe: A Conversational and Longitudinal Benchmark for Utility-Grounded Financial Recommendation](https://mlsanity.com/paper/2602.16990/) (2026)
- [Beyond Teacher Likelihood: Group-Calibrated On-Policy Distillation for Long-Context Reasoning](https://mlsanity.com/paper/2608.19181/) (2026)
- [Improving Table Understanding with LLMs and Entity-Oriented Search](https://mlsanity.com/paper/2508.17028/) (2025)
- [VERA: Identifying and Leveraging Visual Evidence Retrieval Heads in Long-Context Understanding](https://mlsanity.com/paper/2602.10146/) (2026)
- [eC-Tab2Text: Aspect-Based Text Generation from e-Commerce Product Tables](https://mlsanity.com/paper/2502.14820/) (2025)
- [MULTITAT: Benchmarking Multilingual Table-and-Text Question Answering](https://mlsanity.com/paper/2502.17253/) (2025)
- [SUCEA: Reasoning-Intensive Retrieval for Adversarial Fact-checking through Claim Decomposition and Editing](https://mlsanity.com/paper/2506.04583/) (2025)
- [Planning for Success: Exploring LLM Long-term Planning Capabilities in Table Understanding](https://mlsanity.com/paper/2508.17005/) (2025)
- [Merging Continual Pretraining Models for Domain-Specialized LLMs: A Case Study in Finance](https://mlsanity.com/paper/2511.02451/) (2025)
- [Retrieval-Infused Reasoning Sandbox: A Benchmark for Decoupling Retrieval and Reasoning Capabilities](https://mlsanity.com/paper/2601.21937/) (2026)

## Abstract (arXiv)

Recent LLMs have demonstrated remarkable performance in solving exam-like math word problems. However, the degree to which these numerical reasoning skills are effective in real-world scenarios, particularly in expert domains, is still largely unexplored. This paper introduces DocMath-Eval, a comprehensive benchmark specifically designed to evaluate the numerical reasoning capabilities of LLMs in the context of understanding and analyzing specialized documents containing both text and tables. We conduct an extensive evaluation of 48 LLMs with Chain-of-Thought and Program-of-Thought prompting methods, aiming to comprehensively assess the capabilities and limitations of existing LLMs in DocMath-Eval. We found that even the current best-performing system (i.e., GPT-4o) still significantly lags behind human experts in solving complex numerical reasoning problems grounded in long contexts. We believe that DocMath-Eval can serve as a valuable benchmark for evaluating LLMs' capabilities in solving challenging numerical reasoning problems within expert domains.