# HARDMath: A Benchmark Dataset for Challenging Problems in Applied Mathematics

arXiv: https://arxiv.org/abs/2410.09988 · HTML: https://mlsanity.com/paper/2410.09988/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 21 references that are in the map.
- First posted on arXiv 13 Oct 2024.

## Builds on (most cited first, 9 of 9)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms](https://mlsanity.com/paper/1905.13319/) (2019)
- [Mathematical Capabilities of ChatGPT](https://mlsanity.com/paper/2301.13867/) (2023)
- [MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmark](https://mlsanity.com/paper/2405.12209/) (2024)
- [Have LLMs Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Models](https://mlsanity.com/paper/2305.15074/) (2023)
- [ARB: Advanced Reasoning Benchmark for Large Language Models](https://mlsanity.com/paper/2307.13692/) (2023)
- [Is GPT-4 a reliable rater? Evaluating Consistency in GPT-4 Text Ratings](https://mlsanity.com/paper/2308.02575/) (2023)

## Built on by (most cited first, 30 of 32)

- [Towards Robust Mathematical Reasoning](https://mlsanity.com/paper/2511.01846/) (2025)
- [A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challenges](https://mlsanity.com/paper/2412.11936/) (2024)
- [FormalMATH: Benchmarking Formal Mathematical Reasoning of Large Language Models](https://mlsanity.com/paper/2505.02735/) (2025)
- [Physics: Benchmarking Foundation Models on University-Level Physics Problem Solving](https://mlsanity.com/paper/2503.21821/) (2025)
- [FATE: A Formal Benchmark Series for Frontier Algebra of Multiple Difficulty Levels](https://mlsanity.com/paper/2511.02872/) (2025)
- [Large Language Model Reasoning Failures](https://mlsanity.com/paper/2602.06176/) (2026)
- [CriticLean: Critic-Guided Reinforcement Learning for Mathematical Formalization](https://mlsanity.com/paper/2507.06181/) (2025)
- [MatSciBench: Benchmarking the Reasoning Ability of Large Language Models in Materials Science](https://mlsanity.com/paper/2510.12171/) (2025)
- [Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillation](https://mlsanity.com/paper/2606.06712/) (2026)
- [Beyond Single-Value Metrics: Evaluating and Enhancing LLM Unlearning with Cognitive Diagnosis](https://mlsanity.com/paper/2502.13996/) (2025)
- [HARDMath2: A Benchmark for Applied Mathematics Built by Students as Part of a Graduate Class](https://mlsanity.com/paper/2505.11774/) (2025)
- [Learning Efficient and Generalizable Graph Retriever for Knowledge-Graph Question Answering](https://mlsanity.com/paper/2506.09645/) (2025)
- [Do LLMs Overthink Basic Math Reasoning? Benchmarking the Accuracy-Efficiency Tradeoff in Language Models](https://mlsanity.com/paper/2507.04023/) (2025)
- [StyleBench: Evaluating thinking styles in Large Language Models](https://mlsanity.com/paper/2509.20868/) (2025)
- [EngTrace: A Symbolic Benchmark for Verifiable Process Supervision of Engineering Reasoning](https://mlsanity.com/paper/2511.01650/) (2025)
- [CMT-Benchmark: A Benchmark for Condensed Matter Theory Built by Expert Researchers](https://mlsanity.com/paper/2510.05228/) (2025)
- [Domain Specific Benchmarks for Evaluating Multimodal Large Language Models](https://mlsanity.com/paper/2506.12958/) (2025)
- [ECHO-2: A Large-Scale Distributed Rollout Framework for Cost-Efficient Reinforcement Learning](https://mlsanity.com/paper/2602.02192/) (2026)
- [Memory Inception: Latent-Space KV Cache Manipulation for Steering LLMs](https://mlsanity.com/paper/2605.06225/) (2026)
- [ResearchMath-14K: Scaling Research-Level Mathematics via Agents](https://mlsanity.com/paper/2605.28003/) (2026)
- [MAMUT: A Novel Framework for Modifying Mathematical Formulas for the Generation of Specialized Datasets for Language Model Training](https://mlsanity.com/paper/2502.20855/) (2025)
- [Power Law Guided Dynamic Sifting for Efficient Attention](https://mlsanity.com/paper/2506.05300/) (2025)
- [Self-Improvement as Coherence Optimization: A Theoretical Account](https://mlsanity.com/paper/2601.13566/) (2026)
- [Even GPT-5.2 Can't Count to Five: The Case for Zero-Error Horizons in Trustworthy LLMs](https://mlsanity.com/paper/2601.15714/) (2026)
- [Max It or Miss It: Benchmarking LLM On Solving Extremal Problems](https://mlsanity.com/paper/2510.12997/) (2025)
- [MathSight: A Benchmark Exploring Have Vision-Language Models Really Seen in University-Level Mathematical Reasoning?](https://mlsanity.com/paper/2511.23112/) (2025)
- [SPM-Bench: Benchmarking Large Language Models for Scanning Probe Microscopy](https://mlsanity.com/paper/2602.22971/) (2026)
- [The CompMath-MCQ Dataset: Are LLMs Ready for Higher-Level Math?](https://mlsanity.com/paper/2603.03334/) (2026)
- [Identifying Influential N-grams in Confidence Calibration via Regression Analysis](https://mlsanity.com/paper/2604.05757/) (2026)
- [DPrivBench: Benchmarking LLMs' Reasoning for Differential Privacy](https://mlsanity.com/paper/2604.15851/) (2026)

## Abstract (arXiv)

Advanced applied mathematics problems are underrepresented in existing Large Language Model (LLM) benchmark datasets. To address this, we introduce HARDMath, a dataset inspired by a graduate course on asymptotic methods, featuring challenging applied mathematics problems that require analytical approximation techniques. These problems demand a combination of mathematical reasoning, computational tools, and subjective judgment, making them difficult for LLMs. Our framework auto-generates a large number of problems with solutions validated against numerical ground truths. We evaluate both open- and closed-source LLMs on HARDMath-mini, a sub-sampled test set of 366 problems, as well as on 40 word problems formulated in applied science contexts. Even leading closed-source models like GPT-4 achieve only 43.8% overall accuracy with few-shot Chain-of-Thought prompting, and all models demonstrate significantly lower performance compared to results on existing mathematics benchmark datasets. We additionally conduct a detailed error analysis to gain insights into the failure cases of LLMs. These results demonstrate limitations of current LLM performance on advanced graduate-level applied math problems and underscore the importance of datasets like HARDMath to advance mathematical abilities of LLMs.