mlsanityData to 14 Oct 2026

Hard2Verify: A Step-Level Verification Benchmark for Open-Ended Frontier Math

arXiv 2510.13744 ↗First posted 15 Oct 2025cs.AI cs.CL cs.LGcited by 22

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

Large language model (LLM)-based reasoning systems have recently achieved gold medal-level performance in the IMO 2025 competition, writing mathematical proofs where, to receive full credit, each step must be not only correct but also sufficiently supported. To train LLM-based reasoners in such challenging, open-ended settings, strong verifiers capable of catching step-level mistakes are necessary prerequisites. We introduce Hard2Verify, a human-annotated, step-level verification benchmark produced with over 500 hours of human labor. Hard2Verify is designed to rigorously assess step-level verifiers at the frontier: Verifiers must provide step-level annotations or identify the first error in responses generated by frontier LLMs for very recent, challenging, and open-ended math questions. We evaluate 29 generative critics and process reward models, demonstrating that, beyond a few standouts, open-source verifiers lag closed source models. We subsequently analyze what drives poor performance in step-level verification, the impacts of scaling verifier compute, as well as fundamental questions such as self-verification and verification-generation dynamics.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 43 of its 62 references that are in the map. First posted on arXiv 15 Oct 2025.

Builds on

43 of 62 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Reliable Fine-Grained Evaluation of Natural Language Math Proofs14 Oct
  2. 2024FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI7 Nov
  3. 2024Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models10 Oct
  4. 2024Evaluating Mathematical Reasoning Beyond Accuracy8 Apr
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models5 Feb

Other references · 38

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  8. 2023Let's Verify Step by Stepcited by 3,789
  9. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  10. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
  11. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  12. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
26 more
  1. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  2. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  3. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
  4. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  5. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 467
  6. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 441
  7. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoningcited by 415
  8. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 318
  9. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoningcited by 292
  10. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoningcited by 279
  11. 2025Inference-Time Scaling for Generalist Reward Modelingcited by 239
  12. 2023Evaluating the Performance of Large Language Models on GAOKAO Benchmarkcited by 186
  13. 2024Generative Reward Modelscited by 148
  14. 2024On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Taskscited by 142
  15. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 111
  16. 2025PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Modelscited by 99
  17. 2023GPT-4 Doesn't Know It's Wrong: An Analysis of Iterative Prompting for Reasoning Problemscited by 89
  18. 2024How to Evaluate Reward Models for RLHFcited by 88
  19. 2025Multi-Agent Verification: Scaling Test-Time Compute with Multiple Verifierscited by 62
  20. 2023MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluationcited by 57
  21. 2025ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMscited by 46
  22. 2025Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluatorscited by 44
  23. 2025RL Tango: Reinforcing Generator and Verifier Together for Language Reasoningcited by 40
  24. 2025AURORA:Automated Training Framework of Universal Process Reward Models via Ensemble Prompting and Reverse Verificationcited by 23
  25. 2025Variation in Verification: Understanding Verification Dynamics in Large Language Modelscited by 19
  26. 2025Scaling Flaws of Verifier-Guided Search in Mathematical Reasoningcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 20.

22 papers.

Sort:

By area:

  1. 2026Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMscited by 137
  2. 2026AI co-mathematician: Accelerating mathematicians with agentic AIcited by 17
  3. 2025Scaling Generative Verifiers For Natural Language Mathematical Proof Verification And Selectioncited by 9
  4. 2026Do Not Waste Your Rollouts: Recycling Search Experience for Efficient Test-Time Scalingcited by 9
  5. 2026Aletheia: What Makes RLVR For Code Verifiers Tick?cited by 5
  6. 2026Guideline-Grounded Evidence Accumulation for High-Stakes Agent Verificationcited by 5
  7. 2026The Hidden Signal of Verifier Strictness: Controlling and Improving Step-Wise Verification via Selective Latent Steeringcited by 5
  8. 2026Pseudo-Formalization for Automatic Proof Verificationcited by 4
  9. 2025Pessimistic Verification for Open Ended Math Questionscited by 3
  10. 2026Not All Proofs Are Equal: Evaluating LLM Proof Quality Beyond Correctnesscited by 3
  11. 2026Unmasking Reasoning Processes: A Process-aware Benchmark for Evaluating Structural Mathematical Reasoning in LLMscited by 1
  12. 2026Granularity-Regulated Adaptive Computational Efficiency for Optimal Verification in Test-Time Scalingcited by 1
10 more
  1. 2026VeriBound: PAC-Bayesian Generalization Bounds for Process Reward Models Trained with Formal Verification Toolscited by 1
  2. 2026BudgetVerify: Budget-Tiered Verification for Financial QAcited by 1
  3. 2026Embedding Perturbation may Better Reflect Intermediate-Step Uncertainty in LLM Reasoningcited by 0
  4. 2026Mask-Proof: An LLM-based Automated Data Curation Pipeline on Mathematical Proofscited by 0
  5. 2026AdvancedMathBench: A Benchmark Suite for Advanced Mathematical Proof Generation and Verificationcited by 0
  6. 2026FaithSieve: Fine-Grained Evaluation of Math Proofs with Faithful Formal Evidencecited by 0
  7. 2026Learning Strategies to Break Judgescited by 0
  8. 2026RGDT-Bench: Benchmarking LLM Reasoning for Rule-Governed Decisions and Their Justificationscited by 0
  9. 2026LeanPRM: Lean-Grounded Process Supervision for Natural Language Proof Verificationcited by 0
  10. 2026One-Sided Verification Under Limited Expert Reviewcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Do We Need Frontier Models to Verify Mathematical Proofs?
  2. 2025FormalMATH: Benchmarking Formal Mathematical Reasoning of Large Language Models
  3. 2024Not All Votes Count! Programs as Verifiers Improve Self-Consistency of Language Models for Math Reasoning
  4. 2025ProofCompass: Enhancing Specialized Provers with LLM Guidance
  5. 2026EternalMath: A Living Benchmark of Frontier Mathematics that Evolves with Human Discovery
  6. 2025Putnam-AXIOM: A Functional and Static Benchmark for Measuring Higher Level Mathematical Reasoning in LLMs
  7. 2024Improving LLM Reasoning through Scaling Inference Computation with Collaborative Verification
  8. 2025DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning