# HARP: A challenging human-annotated math reasoning benchmark

arXiv: https://arxiv.org/abs/2412.08819 · HTML: https://mlsanity.com/paper/2412.08819/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Placed in Mathematical reasoning, one of 17 directions in LLM reasoning.
- Builds on 22 of its 38 references that are in the map.
- First posted on arXiv 11 Dec 2024.

## Placed in

- LLM reasoning / [Mathematical reasoning](https://mlsanity.com/lr/mathematical-reasoning/)

## Builds on (most cited first, 22 of 22)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Large Language Monkeys: Scaling Inference Compute with Repeated Sampling](https://mlsanity.com/paper/2407.21787/) (2024)
- [STaR: Bootstrapping Reasoning With Reasoning](https://mlsanity.com/paper/2203.14465/) (2022)
- [Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models](https://mlsanity.com/paper/2312.06585/) (2023)
- [Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models](https://mlsanity.com/paper/2410.07985/) (2024)
- [FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI](https://mlsanity.com/paper/2411.04872/) (2024)
- [V-STaR: Training Verifiers for Self-Taught Reasoners](https://mlsanity.com/paper/2402.06457/) (2024)
- [A Careful Examination of Large Language Model Performance on Grade School Arithmetic](https://mlsanity.com/paper/2405.00332/) (2024)
- [Language models show human-like content effects on reasoning tasks](https://mlsanity.com/paper/2207.07051/) (2022)
- [OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text](https://mlsanity.com/paper/2310.06786/) (2023)
- [Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves](https://mlsanity.com/paper/2311.04205/) (2023)
- [RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Fold](https://mlsanity.com/paper/2406.14532/) (2024)
- [Data Diversity Matters for Robust Instruction Tuning](https://mlsanity.com/paper/2311.14736/) (2023)
- [Open-LLM-Leaderboard: From Multi-choice to Open-style Questions for LLMs Evaluation, Benchmark, and Arena](https://mlsanity.com/paper/2406.07545/) (2024)
- [Quantifying Variance in Evaluation Benchmarks](https://mlsanity.com/paper/2406.10229/) (2024)
- [MathOdyssey: Benchmarking Mathematical Problem-Solving Skills in Large Language Models Using Odyssey Math Data](https://mlsanity.com/paper/2406.18321/) (2024)
- [Changing Answer Order Can Decrease MMLU Accuracy](https://mlsanity.com/paper/2406.19470/) (2024)
- [Evaluation data contamination in LLMs: how do we measure it and (when) does it matter?](https://mlsanity.com/paper/2411.03923/) (2024)
- [Keep Guessing? When Considering Inference Scaling, Mind the Baselines](https://mlsanity.com/paper/2410.15466/) (2024)

## Built on by (most cited first, 27 of 27)

- [MathArena: Evaluating LLMs on Uncontaminated Math Competitions](https://mlsanity.com/paper/2505.23281/) (2025)
- [Big-Math: A Large-Scale, High-Quality Math Dataset for Reinforcement Learning in Language Models](https://mlsanity.com/paper/2502.17387/) (2025)
- [CWM: An Open-Weights LLM for Research on Code Generation with World Models](https://mlsanity.com/paper/2510.02387/) (2025)
- [Optimizing Language Models for Inference Time Objectives using Reinforcement Learning](https://mlsanity.com/paper/2503.19595/) (2025)
- [Beyond Verifiable Rewards: Scaling Reinforcement Learning for Language Models to Unverifiable Data](https://mlsanity.com/paper/2503.19618/) (2025)
- [What Characterizes Effective Reasoning? Revisiting Length, Review, and Structure of CoT](https://mlsanity.com/paper/2509.19284/) (2025)
- [Brains vs. Bytes: Evaluating LLM Proficiency in Olympiad Mathematics](https://mlsanity.com/paper/2504.01995/) (2025)
- [Towards High Data Efficiency in Reinforcement Learning with Verifiable Reward](https://mlsanity.com/paper/2509.01321/) (2025)
- [Reliable Fine-Grained Evaluation of Natural Language Math Proofs](https://mlsanity.com/paper/2510.13888/) (2025)
- [GeoGramBench: Benchmarking the Geometric Program Reasoning in Modern LLMs](https://mlsanity.com/paper/2505.17653/) (2025)
- [Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability](https://mlsanity.com/paper/2601.18778/) (2026)
- [MiroMind-M1: An Open-Source Advancement in Mathematical Reasoning via Context-Aware Multi-Stage Policy Optimization](https://mlsanity.com/paper/2507.14683/) (2025)
- [EngiBench: A Benchmark for Evaluating Large Language Models on Engineering Problem Solving](https://mlsanity.com/paper/2509.17677/) (2025)
- [A Survey of Deep Learning for Geometry Problem Solving](https://mlsanity.com/paper/2507.11936/) (2025)
- [RefGrader: Automated Grading of Mathematical Competition Proofs using Agentic Workflows](https://mlsanity.com/paper/2510.09021/) (2025)
- [ConceptMoE: Adaptive Token-to-Concept Compression for Implicit Compute Allocation](https://mlsanity.com/paper/2601.21420/) (2026)
- [CombiGraph-Vis: A Curated Multimodal Olympiad Benchmark for Discrete Mathematical Reasoning](https://mlsanity.com/paper/2510.27094/) (2025)
- [DialDefer: A Framework for Detecting and Mitigating LLM Dialogic Deference](https://mlsanity.com/paper/2601.10896/) (2026)
- [Making Expert Reasoning Learnable with Self-Distillation](https://mlsanity.com/paper/2602.02405/) (2026)
- [Strategy Executability in Mathematical Reasoning: Leveraging Human-Model Differences for Effective Guidance](https://mlsanity.com/paper/2602.22583/) (2026)
- [RLVR Datasets and Where to Find Them: Tracing Data Lineage for Better Training Data](https://mlsanity.com/paper/2605.26971/) (2026)
- [EasyMath: A 0-shot Math Benchmark for SLMs](https://mlsanity.com/paper/2505.14852/) (2025)
- [The CompMath-MCQ Dataset: Are LLMs Ready for Higher-Level Math?](https://mlsanity.com/paper/2603.03334/) (2026)
- [Re2Math: Benchmarking Theorem Retrieval in Research-Level Mathematics](https://mlsanity.com/paper/2605.09012/) (2026)
- [CrowdMath: A Dataset of Crowdsourced Mathematical Research Discussions](https://mlsanity.com/paper/2606.06526/) (2026)
- [COMPASS: Finding Where Reasoning Lives in Language Models](https://mlsanity.com/paper/2610.07469/) (2026)
- [One Model, Multiple Compute Budgets: Multi-Ratio ConceptMoE for Flexible Compute Across Turns](https://mlsanity.com/paper/2610.14074/) (2026)

## Abstract (arXiv)

Math reasoning is becoming an ever increasing area of focus as we scale large language models. However, even the previously-toughest evals like MATH are now close to saturated by frontier models (90.0% for o1-mini and 86.5% for Gemini 1.5 Pro). We introduce HARP, Human Annotated Reasoning Problems (for Math), consisting of 5,409 problems from the US national math competitions (A(J)HSME, AMC, AIME, USA(J)MO). Of these, 4,780 have answers that are automatically check-able (with libraries such as SymPy). These problems range six difficulty levels, with frontier models performing relatively poorly on the hardest bracket of 197 problems (average accuracy 41.1% for o1-mini, and 9.6% for Gemini 1.5 Pro). Our dataset also features multiple choices (for 4,110 problems) and an average of two human-written, ground-truth solutions per problem, offering new avenues of research that we explore briefly. We report evaluations for many frontier models and share some interesting analyses, such as demonstrating that frontier models across families intrinsically scale their inference-time compute for more difficult problems. Finally, we open source all code used for dataset construction (including scraping) and all code for evaluation (including answer checking) to enable future research at: https://github.com/aadityasingh/HARP.