mlsanityData to 14 Oct 2026

Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiad

arXiv 2503.21934 ↗First posted 27 Mar 2025cs.CLcited by 74

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

Recent math benchmarks for large language models (LLMs) such as MathArena indicate that state-of-the-art reasoning models achieve impressive performance on mathematical competitions like AIME, with the leading model, Gemini-2.5-Pro, achieving scores comparable to top human competitors. However, these benchmarks evaluate models solely based on final numerical answers, neglecting rigorous reasoning and proof generation which are essential for real-world mathematical tasks. To address this, we introduce a comprehensive evaluation of full-solution reasoning for challenging mathematical problems. Using expert human annotators, we evaluated several state-of-the-art reasoning models on the six problems from the 2025 USAMO within hours of their release. Our results reveal that all tested models struggled significantly: only Gemini-2.5-Pro achieves a non-trivial score of 25%, while all other models achieve less than 5%. Through detailed analysis of reasoning traces, we identify the most common failure modes and find several unwanted artifacts arising from the optimization strategies employed during model training. Overall, our results suggest that current LLMs are inadequate for rigorous mathematical reasoning tasks, highlighting the need for substantial improvements in reasoning and proof generation capabilities.

Key facts. Cited by 74 papers in the map (data through 14 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 8 of its 14 references that are in the map. First posted on arXiv 27 Mar 2025.

Builds on

8 of 14 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025MathConstruct: Challenging LLM Reasoning with Constructive Proofs14 Feb
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models5 Feb

Other references · 6

  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  2. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  3. 2021MiniF2F: a cross-system benchmark for formal Olympiad-level mathematicscited by 391
  4. 2023Mathematical Capabilities of ChatGPTcited by 268
  5. 2024PutnamBench: Evaluating Neural Theorem-Provers on the Putnam Mathematical Competitioncited by 165
  6. 2023FIMO: A Challenge Formal Dataset for Automated Theorem Provingcited by 55

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 53; 2026: 21.

74 papers.

Sort:

By area:

  1. 2025MathArena: Evaluating LLMs on Uncontaminated Math Competitionscited by 297
  2. 2025Goedel-Prover-V2: Scaling Formal Theorem Proving with Scaffolded Data Synthesis and Self-Correctioncited by 154
  3. 2025Phi-4-reasoning Technical Reportcited by 151
  4. 2026Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMscited by 137
  5. 2025Seed-Prover: Deep and Broad Reasoning for Automated Theorem Provingcited by 101
  6. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 93
  7. 2025Winning Gold at IMO 2025 with a Model-Agnostic Verification-and-Refinement Pipelinecited by 82
  8. 2026AutoHarness: improving LLM agents by automatically synthesizing a code harnesscited by 76
  9. 2025PHYBench: Holistic Evaluation of Physical Perception and Reasoning in Large Language Modelscited by 61
  10. 2025Hilbert: Recursively Building Formal Proofs with Informal Reasoningcited by 60
  11. 2025AMO-Bench: Large Language Models Still Struggle in High School Math Competitionscited by 38
  12. 2025No Free Lunch: Rethinking Internal Feedback for LLM Reasoningcited by 37
38 more
  1. 2025RealMath: A Continuous Benchmark for Evaluating Language Models on Research-Level Mathematicscited by 31
  2. 2025The Open Proof Corpus: A Large-Scale Study of LLM-Generated Mathematical Proofscited by 30
  3. 2025BrokenMath: A Benchmark for Sycophancy in Theorem Proving with LLMscited by 27
  4. 2025When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Researchcited by 26
  5. 2025Solving Inequality Proofs with Large Language Modelscited by 21
  6. 2025RLAD: Training LLMs to Discover Abstractions for Solving Reasoning Problemscited by 21
  7. 2025Mathematical Proof as a Litmus Test: Revealing Failure Modes of Advanced Large Reasoning Modelscited by 20
  8. 2025Reliable Fine-Grained Evaluation of Natural Language Math Proofscited by 18
  9. 2025Reviving DSP for Advanced Theorem Proving in the Era of Reasoning Modelscited by 17
  10. 2025SSRL: Self-Search Reinforcement Learningcited by 14
  11. 2025Value-Guided Search for Efficient Chain-of-Thought Reasoningcited by 13
  12. 2025IMProofBench: Benchmarking AI on Research-Level Mathematical Proof Generationcited by 13
  13. 2025ROC-n-reroll: How verifier imperfection affects test-time scalingcited by 12
  14. 2025Test-Time Scaling in Reasoning Models Is Not Effective for Knowledge-Intensive Tasks Yetcited by 11
  15. 2025Right Is Not Enough: The Pitfalls of Outcome Supervision in Training LLMs for Math Reasoningcited by 10
  16. 2025Ineq-Comp: Benchmarking Human-Intuitive Compositional Reasoning in Automated Theorem Proving on Inequalitiescited by 8
  17. 2025An Investigation of Robustness of LLMs in Mathematical Reasoning: Benchmarking with Mathematically-Equivalent Transformation of Advanced Mathematical Problemscited by 8
  18. 2025Are Large Language Models Robust in Understanding Code Against Semantics-Preserving Mutations?cited by 7
  19. 2026BAS: A Decision-Theoretic Approach to Evaluating Large Language Model Confidencecited by 7
  20. 2025Has the Creativity of Large-Language Models peaked? An analysis of inter- and intra-LLM variabilitycited by 6
  21. 2025Seesaw: Accelerating Training by Balancing Learning Rate and Batch Size Schedulingcited by 6
  22. 2025HARDMath2: A Benchmark for Applied Mathematics Built by Students as Part of a Graduate Classcited by 5
  23. 2025Who's Laughing Now? An Overview of Computational Humour Generation and Explanationcited by 5
  24. 2025Beyond Reasoning Gains: Mitigating General-Capability Forgetting in Large Reasoning Modelscited by 5
  25. 2025Thinking Machines: Mathematical Reasoning in the Age of LLMscited by 4
  26. 2025Mathematical Computation and Reasoning Errors by Large Language Modelscited by 4
  27. 2025EpidemIQs: Prompt-to-Paper LLM Agents for Epidemic Modeling and Analysiscited by 4
  28. 2025PRISM-Physics: Causal DAG-Based Process Evaluation for Physics Reasoningcited by 4
  29. 2025RefGrader: Automated Grading of Mathematical Competition Proofs using Agentic Workflowscited by 4
  30. 2025Are Language Models Efficient Reasoners? A Perspective from Logic Programmingcited by 4
  31. 2026Geometry of Reason: Spectral Signatures of Valid Mathematical Reasoningcited by 4
  32. 2025Sustainability via LLM Right-sizingcited by 3
  33. 2025Using Reasoning Models to Generate Search Heuristics that Solve Open Instances of Combinatorial Design Problemscited by 3
  34. 2025ChemLabs on ChemO: A Multi-Agent System for Multimodal Reasoning on IChO 2025cited by 3
  35. 2026Student Guides Teacher: Weak-to-Strong Inference via Spectral Orthogonal Explorationcited by 3
  36. 2026Exposing Weaknesses of Large Reasoning Models through Graph Algorithm Problemscited by 3
  37. 2026Do We Need Frontier Models to Verify Mathematical Proofs?cited by 3
  38. 2026QED-Nano: Teaching a Tiny Model to Prove Hard Theoremscited by 3

50 most cited of 74 All 74 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Brains vs. Bytes: Evaluating LLM Proficiency in Olympiad Mathematics
  2. 2025EEFSUVA: A New Mathematical Olympiad Benchmark
  3. 2025Evaluating the Reasoning Abilities of LLMs on Underrepresented Mathematics Competition Problems
  4. 2025Evaluation of LLMs for mathematical problem solving
  5. 2025RIMO: An Easy-to-Evaluate, Hard-to-Solve Olympiad Benchmark for Advanced Mathematical Reasoning
  6. 2026The CompMath-MCQ Dataset: Are LLMs Ready for Higher-Level Math?
  7. 2026Mathematical Reasoning in Large Language Models: Benchmarks, Architectures, Evaluation, and Open Challenges
  8. 2024MathOdyssey: Benchmarking Mathematical Problem-Solving Skills in Large Language Models Using Odyssey Math Data