mlsanityData to 7 Oct 2026

OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems

arXiv 2402.14008 ↗First posted 21 Feb 2024cs.CLcited by 1,099

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Recent advancements have seen Large Language Models (LLMs) and Large Multimodal Models (LMMs) surpassing general human capabilities in various tasks, approaching the proficiency level of human experts across multiple domains. With traditional benchmarks becoming less challenging for these models, new rigorous challenges are essential to gauge their advanced abilities. In this work, we present OlympiadBench, an Olympiad-level bilingual multimodal scientific benchmark, featuring 8,476 problems from Olympiad-level mathematics and physics competitions, including the Chinese college entrance exam. Each problem is detailed with expert-level annotations for step-by-step reasoning. Evaluating top-tier models on OlympiadBench, we implement a comprehensive assessment methodology to accurately evaluate model responses. Notably, the best-performing model, GPT-4V, attains an average score of 17.97% on OlympiadBench, with a mere 10.74% in physics, highlighting the benchmark rigor and the intricacy of physical reasoning. Our analysis orienting GPT-4V points out prevalent issues with hallucinations, knowledge omissions, and logical fallacies. We hope that our challenging benchmark can serve as a valuable resource for helping future AGI research endeavors. The data and evaluation code are available at urlhttps://github.com/OpenBMB/OlympiadBench

Key facts. Cited by 1,099 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 66 references that are in the map. First posted on arXiv 21 Feb 2024.

Builds on

32 of 66 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct

Other references · 31

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  7. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  8. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  9. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  10. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  11. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  12. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
19 more
  1. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  2. 2021MiniF2F: a cross-system benchmark for formal Olympiad-level mathematicscited by 380
  3. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 376
  4. 2022Draft, Sketch, and Prove: Guiding Formal Theorem Provers with Informal Proofscited by 278
  5. 2023Mathematical Capabilities of ChatGPTcited by 263
  6. 2021GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoningcited by 224
  7. 2023TheoremQA: A Theorem-driven Question Answering datasetcited by 189
  8. 2023Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verificationcited by 166
  9. 2022UniGeo: Unifying Geometry Logical Reasoning via Reformulating Mathematical Expressioncited by 137
  10. 2022Large Language Models Meet NL2Code: A Surveycited by 135
  11. 2022A Survey of Deep Learning for Mathematical Reasoningcited by 124
  12. 2023SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Researchcited by 112
  13. 2023CMATH: Can Your Language Model Pass Chinese Elementary School Math Test?cited by 89
  14. 2021NaturalProofs: Mathematical Theorem Proving in Natural Languagecited by 80
  15. 2023Evaluating Language Models for Mathematics through Interactionscited by 69
  16. 2023A Brief Report on LawGPT 1.0: A Virtual Legal Assistant Based on GPT-3cited by 46
  17. 2024CMMU: A Benchmark for Chinese Multi-modal Multi-type Question Understanding and Reasoningcited by 31
  18. 2023Transfer Knowledge from Natural Language to Electrocardiography: Can We Detect Cardiovascular Disease Through Language Models?cited by 21
  19. 2023AI for Mathematics: A Cognitive Science Perspectivecited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 39; 2025: 540; 2026: 520.

1,099 papers.

Sort:

By area:

  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  2. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  3. 2025s1: Simple test-time scalingcited by 1,311
  4. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  5. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  6. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 948
  7. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 540
  8. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 505
  9. 2025LIMO: Less is More for Reasoningcited by 488
  10. 2025Process Reinforcement through Implicit Rewardscited by 478
  11. 2025Humanity's Last Examcited by 467
  12. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 463
38 more
  1. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  2. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoningcited by 402
  3. 2025L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learningcited by 381
  4. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 344
  5. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 323
  6. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  7. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 308
  8. 2025MM-Eureka: Exploring the Frontiers of Multimodal Reasoning with Rule-based Reinforcement Learningcited by 294
  9. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 288
  10. 2024Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Modelscited by 277
  11. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoningcited by 277
  12. 2025MathArena: Evaluating LLMs on Uncontaminated Math Competitionscited by 275
  13. 2025Learning to Reason under Off-Policy Guidancecited by 266
  14. 2025LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RLcited by 244
  15. 2024FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AIcited by 227
  16. 2025Dynamic Early Exit in Reasoning Modelscited by 217
  17. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  18. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 210
  19. 2025ThinkPrune: Pruning Long Chain-of-Thought of LLMs via Reinforcement Learningcited by 187
  20. 2025Reasoning Models Can Be Effective Without Thinkingcited by 182
  21. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  22. 2026GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimizationcited by 175
  23. 2025ToRL: Scaling Tool-Integrated RLcited by 166
  24. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoningcited by 160
  25. 2024Accessing GPT-4 level Mathematical Olympiad Solutions via Monte Carlo Tree Self-refine with LLaMa-3 8Bcited by 156
  26. 2025LLaDA2.0: Scaling Up Diffusion Language Models to 100Bcited by 154
  27. 2026Entropy-Aware On-Policy Distillation of Language Modelscited by 151
  28. 2025Scaling Latent Reasoning via Looped Language Modelscited by 145
  29. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforcecited by 141
  30. 2025Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmarkcited by 132
  31. 2025General-Reasoner: Advancing LLM Reasoning Across All Domainscited by 132
  32. 2026Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixescited by 122
  33. 2026SkillOpt: Executive Strategy for Self-Evolving Agent Skillscited by 122
  34. 2025REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewardscited by 121
  35. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectificationcited by 121
  36. 2025Process Reward Models That Thinkcited by 118
  37. 2025OctoThinker: Mid-training Incentivizes Reinforcement Learning Scalingcited by 110
  38. 2025Small Models Struggle to Learn from Strong Reasonerscited by 109

50 most cited of 1,099 All 1,099 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026PhysElite: How Far Are LLMs from Solving Olympiad-Level Physics Problems?
  2. 2025EEFSUVA: A New Mathematical Olympiad Benchmark
  3. 2024Can Language Models Solve Olympiad Programming?
  4. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models
  5. 2025LiveOIBench: Can Large Language Models Outperform Human Contestants in Informatics Olympiads?
  6. 2025LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming?
  7. 2023Have LLMs Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Models
  8. 2025Multi-Physics: A Comprehensive Benchmark for Multimodal LLMs Reasoning on Chinese Multi-Subject Physics Problems