mlsanityData to 7 Oct 2026

RefGrader: Automated Grading of Mathematical Competition Proofs using Agentic Workflows

arXiv 2510.09021 ↗First posted 10 Oct 2025cs.AI cs.LGcited by 3

On the map: LLM agents and tool use → Educational agents and math problem solving

Abstract (arXiv)

State-of-the-art (SOTA) LLMs have progressed from struggling on proof-based Olympiad problems to solving most of the IMO 2025 problems, with leading systems reportedly handling 5 of 6 problems. Given this progress, we assess how well these models can grade proofs: detecting errors, judging their severity, and assigning fair scores beyond binary correctness. We study proof-analysis capabilities using a corpus of 90 Gemini 2.5 Pro-generated solutions that we grade on a 1-4 scale with detailed error annotations, and on MathArena solution sets for IMO/USAMO 2025 scored on a 0-7 scale. Our analysis shows that models can reliably flag incorrect (including subtly incorrect) solutions but exhibit calibration gaps in how partial credit is assigned. To address this, we introduce agentic workflows that extract and analyze reference solutions and automatically derive problem-specific rubrics for a multi-step grading process. We instantiate and compare different design choices for the grading workflows, and evaluate their trade-offs. Across our annotated corpus and MathArena, our proposed workflows achieve higher agreement with human grades and more consistent handling of partial credit across metrics. We release all code, data, and prompts/logs to facilitate future research.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Educational agents and math problem solving, one of 16 directions in LLM agents and tool use. Builds on 52 of its 85 references that are in the map. First posted on arXiv 10 Oct 2025.

Builds on

52 of 85 references.

References · 52

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  7. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  8. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  9. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  10. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  11. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 533
  12. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
38 more
  1. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  2. 2021MiniF2F: a cross-system benchmark for formal Olympiad-level mathematicscited by 380
  3. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  4. 2024Large Language Models for Mathematical Reasoning: Progresses and Challengescited by 283
  5. 2024Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Modelscited by 277
  6. 2025MathArena: Evaluating LLMs on Uncontaminated Math Competitionscited by 275
  7. 2025DeepSeek-Prover-V2: Advancing Formal Mathematical Reasoning via Reinforcement Learning for Subgoal Decompositioncited by 270
  8. 2023Mathematical Capabilities of ChatGPTcited by 263
  9. 2024Chain-of-Thought Reasoning Without Promptingcited by 208
  10. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmeticcited by 188
  11. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 146
  12. 2025Goedel-Prover-V2: Scaling Formal Theorem Proving with Scaffolded Data Synthesis and Self-Correctioncited by 140
  13. 2023How well do Large Language Models perform in Arithmetic tasks?cited by 130
  14. 2024GLoRe: When, Where, and How to Improve LLM Reasoning via Global and Local Refinementscited by 101
  15. 2023CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotationcited by 93
  16. 2024LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Textscited by 81
  17. 2024Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gapcited by 79
  18. 2025Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiadcited by 72
  19. 2024Evaluating Mathematical Reasoning Beyond Accuracycited by 69
  20. 2022Parsel: Algorithmic Reasoning with Language Models by Composing Decompositionscited by 62
  21. 2024Language Models are Hidden Reasoners: Unlocking Latent Reasoning Capabilities via Self-Rewardingcited by 53
  22. 2024MathOdyssey: Benchmarking Mathematical Problem-Solving Skills in Large Language Models Using Odyssey Math Datacited by 51
  23. 2024Is Your Model Really A Good Math Reasoner? Evaluating Mathematical Reasoning with Checklistcited by 48
  24. 2024CHAMP: A Competition-level Dataset for Fine-Grained Analyses of LLMs' Mathematical Reasoning Capabilitiescited by 43
  25. 2024Not All LLM Reasoners Are Created Equalcited by 29
  26. 2025Rubric Is All You Need: Enhancing LLM-based Code Evaluation With Question-Specific Rubricscited by 29
  27. 2024LeanReasoner: Boosting Complex Logical Reasoning with Leancited by 28
  28. 2025The Open Proof Corpus: A Large-Scale Study of LLM-Generated Mathematical Proofscited by 27
  29. 2024HARP: A challenging human-annotated math reasoning benchmarkcited by 25
  30. 2024From Calculation to Adjudication: Examining LLM judges on Mathematical Reasoning Taskscited by 22
  31. 2025Brains vs. Bytes: Evaluating LLM Proficiency in Olympiad Mathematicscited by 22
  32. 2025ExpertLongBench: Benchmarking Language Models on Expert-Level Long-Form Generation Tasks with Structured Checklistscited by 22
  33. 2025Mathematical Proof as a Litmus Test: Revealing Failure Modes of Advanced Large Reasoning Modelscited by 18
  34. 2024Evaluating Large Vision-and-Language Models on Children's Mathematical Olympiadscited by 16
  35. 2024A LLM-Powered Automatic Grading Framework with Human-Level Guidelines Optimizationcited by 16
  36. 2024PiCO: Peer Review in LLMs based on the Consistency Optimizationcited by 14
  37. 2024Dual Instruction Tuning with Large Language Models for Mathematical Reasoningcited by 10
  38. 2024Ranking Large Language Models without Ground Truthcited by 7

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2026Not All Proofs Are Equal: Evaluating LLM Proof Quality Beyond Correctnesscited by 3
  2. 2025Pessimistic Verification for Open Ended Math Questionscited by 2
  3. 2026Confirming Correct, Missing the Rest: LLM Tutoring Agents Struggle Where Feedback Matters Mostcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025CombiGraph-Vis: A Curated Multimodal Olympiad Benchmark for Discrete Mathematical Reasoning
  2. 2026RMA: an Agentic System for Research-Level Mathematical Problems
  3. 2026Riemann-Bench: A Benchmark for Moonshot Mathematics
  4. 2025RIMO: An Easy-to-Evaluate, Hard-to-Solve Olympiad Benchmark for Advanced Mathematical Reasoning
  5. 2026AdvancedMathBench: A Benchmark Suite for Advanced Mathematical Proof Generation and Verification
  6. 2026ProofCouncil: An LLM Agent for Solving Open Mathematical Problems
  7. 2026LEAP: Supercharging LLMs for Formal Mathematics with Agentic Frameworks
  8. 2026Escaping the Cognitive Well: Efficient Competition Math with Off-the-Shelf Models