mlsanityData to 14 Oct 2026

Brains vs. Bytes: Evaluating LLM Proficiency in Olympiad Mathematics

arXiv 2504.01995 ↗First posted 1 Apr 2025cs.AI cs.LGcited by 24

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

Recent advances in large language models (LLMs) have shown impressive progress in mathematical reasoning tasks. However, current evaluation benchmarks predominantly focus on the accuracy of final answers, often overlooking the crucial logical rigor for mathematical problem solving. The claim that state-of-the-art LLMs can solve Math Olympiad-level problems requires closer examination. To explore this, we conducted both qualitative and quantitative human evaluations of proofs generated by LLMs, and developed a schema for automatically assessing their reasoning capabilities. Our study reveals that current LLMs fall significantly short of solving challenging Olympiad-level problems and frequently fail to distinguish correct mathematical reasoning from clearly flawed solutions. Our analyses demonstrate that the occasional correct final answers provided by LLMs often result from pattern recognition or heuristic shortcuts rather than genuine mathematical reasoning. These findings underscore the substantial gap between LLM performance and human expertise in advanced mathematical reasoning and highlight the importance of developing benchmarks that prioritize the soundness of the reasoning used to arrive at an answer rather than the mere correctness of the final answers.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 49 of its 56 references that are in the map. First posted on arXiv 1 Apr 2025.

Builds on

49 of 56 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024HARP: A challenging human-annotated math reasoning benchmark11 Dec
  2. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models7 Oct
  3. 2024Not All LLM Reasoners Are Created Equal2 Oct
  4. 2024Is Your Model Really A Good Math Reasoner? Evaluating Mathematical Reasoning with Checklist11 Jul
  5. 2024Evaluating Large Vision-and-Language Models on Children's Mathematical Olympiads22 Jun
  6. 2024OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for Superintelligent AI18 Jun
  7. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmetic1 May
  8. 2024Evaluating Mathematical Reasoning Beyond Accuracy8 Apr
  9. 2024Dual Instruction Tuning with Large Language Models for Mathematical Reasoning27 Mar
  10. 2024LeanReasoner: Boosting Complex Logical Reasoning with Lean20 Mar
  11. 2024Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gap29 Feb
  12. 2024Large Language Models for Mathematical Reasoning: Progresses and Challenges31 Jan
3 more
  1. 2024CHAMP: A Competition-level Dataset for Fine-Grained Analyses of LLMs' Mathematical Reasoning Capabilities13 Jan
  2. 2023ARB: Advanced Reasoning Benchmark for Large Language Models25 Jul
  3. 2023How well do Large Language Models perform in Arithmetic tasks?16 Mar

Other references · 34

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  7. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  8. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  9. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  10. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  11. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  12. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 810
22 more
  1. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  2. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  3. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 662
  4. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  5. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  6. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  7. 2021MiniF2F: a cross-system benchmark for formal Olympiad-level mathematicscited by 391
  8. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 368
  9. 2023Mathematical Capabilities of ChatGPTcited by 268
  10. 2024Chain-of-Thought Reasoning Without Promptingcited by 210
  11. 2023LM vs LM: Detecting Factual Errors via Cross Examinationcited by 156
  12. 2023Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answeringcited by 151
  13. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 149
  14. 2023CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotationcited by 94
  15. 2022Parsel: Algorithmic Reasoning with Language Models by Composing Decompositionscited by 64
  16. 2024Language Models are Hidden Reasoners: Unlocking Latent Reasoning Capabilities via Self-Rewardingcited by 55
  17. 2024MathOdyssey: Benchmarking Mathematical Problem-Solving Skills in Large Language Models Using Odyssey Math Datacited by 52
  18. 2024From Calculation to Adjudication: Examining LLM judges on Mathematical Reasoning Taskscited by 22
  19. 2024PiCO: Peer Review in LLMs based on the Consistency Optimizationcited by 14
  20. 2024Ranking Large Language Models without Ground Truthcited by 8
  21. 2024A Demonstration of Adaptive Collaboration of Large Language Models for Medical Decision-Makingcited by 5
  22. 2024JurEE not Judges: safeguarding llm interactions with small, specialised Encoder Ensemblescited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 15; 2026: 9.

24 papers.

Sort:

By area:

  1. 2025MathArena: Evaluating LLMs on Uncontaminated Math Competitionscited by 297
  2. 2026Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMscited by 137
  3. 2025Winning Gold at IMO 2025 with a Model-Agnostic Verification-and-Refinement Pipelinecited by 82
  4. 2025Hilbert: Recursively Building Formal Proofs with Informal Reasoningcited by 60
  5. 2026Fast and Effective On-policy Distillation from Reasoning Prefixescited by 53
  6. 2025The Open Proof Corpus: A Large-Scale Study of LLM-Generated Mathematical Proofscited by 30
  7. 2025BrokenMath: A Benchmark for Sycophancy in Theorem Proving with LLMscited by 27
  8. 2025IMProofBench: Benchmarking AI on Research-Level Mathematical Proof Generationcited by 13
  9. 2025Right Is Not Enough: The Pitfalls of Outcome Supervision in Training LLMs for Math Reasoningcited by 10
  10. 2025Scaling Generative Verifiers For Natural Language Mathematical Proof Verification And Selectioncited by 9
  11. 2025Knowledge Augmented Complex Problem Solving with Large Language Models: A Surveycited by 6
  12. 2025Can LLMs Refuse Questions They Do Not Know? Measuring Knowledge-Aware Refusal in Factual Taskscited by 6
12 more
  1. 2025RefGrader: Automated Grading of Mathematical Competition Proofs using Agentic Workflowscited by 4
  2. 2025Hop, Skip, and Overthink: Diagnosing Why Reasoning Models Fumble during Multi-Hop Analysiscited by 3
  3. 2025ChemLabs on ChemO: A Multi-Agent System for Multimodal Reasoning on IChO 2025cited by 3
  4. 2026QED-Nano: Teaching a Tiny Model to Prove Hard Theoremscited by 3
  5. 2026Not All Proofs Are Equal: Evaluating LLM Proof Quality Beyond Correctnesscited by 3
  6. 2025Putnam-like dataset summary: LLMs as mathematical competition contestantscited by 2
  7. 2026Right Answer, Wrong Method: Shortcut Hacking Misleads the Evaluation of LLM Reasoning on Frontier Science Benchmarkscited by 2
  8. 2025RCScore: Quantifying Response Consistency in Large Language Modelscited by 1
  9. 2026Strategy Executability in Mathematical Reasoning: Leveraging Human-Model Differences for Effective Guidancecited by 1
  10. 2026STAR-P'olyaMath: Multi-Agent Reasoning under Persistent Meta-Strategic Supervisioncited by 1
  11. 2026Stress-Testing the Reasoning Competence of LLMs With Proofs Under Minimal Formalismcited by 0
  12. 2026Splicing Hints into Reasoning Traces: Learning from Demonstrator Failurescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiad
  2. 2025EEFSUVA: A New Mathematical Olympiad Benchmark
  3. 2025Evaluating the Reasoning Abilities of LLMs on Underrepresented Mathematics Competition Problems
  4. 2025AMO-Bench: Large Language Models Still Struggle in High School Math Competitions
  5. 2025Proving Olympiad Inequalities by Synergizing LLMs and Symbolic Reasoning
  6. 2025CogMath: Assessing LLMs' Authentic Mathematical Ability from a Human Cognitive Perspective
  7. 2025An Investigation of Robustness of LLMs in Mathematical Reasoning: Benchmarking with Mathematically-Equivalent Transformation of Advanced Mathematical Problems
  8. 2026The CompMath-MCQ Dataset: Are LLMs Ready for Higher-Level Math?