mlsanityData to 7 Oct 2026

MathHay: An Automated Benchmark for Long-Context Mathematical Reasoning in LLMs

arXiv 2410.04698 ↗First posted 7 Oct 2024cs.CLcited by 13

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

Recent large language models (LLMs) have demonstrated versatile capabilities in long-context scenarios. Although some recent benchmarks have been developed to evaluate the long-context capabilities of LLMs, there is a lack of benchmarks evaluating the mathematical reasoning abilities of LLMs over long contexts, which is crucial for LLMs' application in real-world scenarios. In this paper, we introduce MathHay, an automated benchmark designed to assess the long-context mathematical reasoning capabilities of LLMs. Unlike previous benchmarks like Needle in a Haystack, which focus primarily on information retrieval within long texts, MathHay demands models with both information-seeking and complex mathematical reasoning abilities. We conduct extensive experiments on MathHay to assess the long-context mathematical reasoning abilities of eight top-performing LLMs. Even the best-performing model, Gemini-1.5-Pro-002, still struggles with mathematical reasoning over long contexts, achieving only 51.26% accuracy at 128K tokens. This highlights the significant room for improvement on the MathHay benchmark.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 29 of its 36 references that are in the map. First posted on arXiv 7 Oct 2024.

Builds on

29 of 36 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023DocMath-Eval: Evaluating Math Reasoning Capabilities of LLMs in Understanding Long and Specialized Documents16 Nov
  2. 2022Lila: A Unified Benchmark for Mathematical Reasoning31 Oct

Other references · 27

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  7. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  8. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  9. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  10. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  11. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  12. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
15 more
  1. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
  2. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  3. 2021Efficient Attentions for Long Document Summarizationcited by 342
  4. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 279
  5. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 242
  6. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  7. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 211
  8. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 175
  9. 2024Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Modelscited by 150
  10. 2024Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QAcited by 121
  11. 2024Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systemscited by 87
  12. 2024DocFinQA: A Long-Context Financial Reasoning Datasetcited by 59
  13. 2023BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Modelscited by 53
  14. 2024Long Code Arena: a Set of Benchmarks for Long-Context Code Modelscited by 48
  15. 2024NeedleBench: Evaluating LLM Retrieval and Reasoning Across Varying Information Densitiescited by 32

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 6; 2026: 6.

13 papers.

Sort:

By area:

  1. 2024A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challengescited by 55
  2. 2025Oolong: Evaluating Long Context Reasoning and Aggregation Capabilitiescited by 33
  3. 2026Benchmark Test-Time Scaling of General LLM Agentscited by 24
  4. 2025Zephyrus: An Agentic Framework for Weather Sciencecited by 8
  5. 2026AgentLongBench: A Controllable Long Benchmark For Long-Contexts Agents via Environment Rolloutscited by 8
  6. 2026ConvexBench: Can LLMs Recognize Convex Functions?cited by 3
  7. 2026Do Reasoning LLMs Refuse What They Infer in Long Contexts?cited by 3
  8. 2025SCALAR: Scientific Citation-based Live Assessment of Long-context Academic Reasoningcited by 1
  9. 2025Harnessing the Unseen: The Hidden Influence of Intrinsic Knowledge in Long-Context Language Modelscited by 1
  10. 2025SciTrek: Evaluating and Improving Long-Context Numerical Reasoning over Scientific Articlescited by 1
  11. 2025MatheMagic: Generating Dynamic Mathematics Benchmarks Robust to Memorizationcited by 1
  12. 2026Almieyar-Oryx-BloomBench: A Bilingual Multimodal Benchmark for Cognitively Informed Evaluation of Vision-Language Modelscited by 1
1 more
  1. 2026Messier: A High-Resolution Corpus for Cross-Benchmark Agent Evaluationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Benchmarking Large Language Models for Math Reasoning Tasks
  2. 2026Mathematical Reasoning in Large Language Models: Benchmarks, Architectures, Evaluation, and Open Challenges
  3. 2025Do LLMs Overthink Basic Math Reasoning? Benchmarking the Accuracy-Efficiency Tradeoff in Language Models
  4. 2025Large Language Models and Mathematical Reasoning Failures
  5. 2025Can LLMs textitunderstand Math? -- Exploring the Pitfalls in Mathematical Reasoning
  6. 2024LLMs Are Not Intelligent Thinkers: Introducing Mathematical Topic Tree Benchmark for Comprehensive Evaluation of LLMs
  7. 2025GSM-Infinite: How Do Your LLMs Behave over Infinitely Increasing Context Length and Reasoning Complexity?
  8. 2025Mathematical Reasoning in Large Language Models: Assessing Logical and Arithmetic Errors across Wide Numerical Ranges