# Evaluating GPT- and Reasoning-based Large Language Models on Physics Olympiad Problems: Surpassing Human Performance and Implications for Educational Assessment arXiv: https://arxiv.org/abs/2505.09438 ยท HTML: https://mlsanity.com/paper/2505.09438/ ## Key facts - Cited by 7 papers in the map (data through 14 Oct 2026). - Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. - Builds on 27 of its 105 references that are in the map. - First posted on arXiv 14 May 2025. ## Placed in - LLM reasoning / [Mathematical reasoning](https://mlsanity.com/lr/mathematical-reasoning/) ## Builds on (most cited first, 27 of 27) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022) - [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022) - [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023) - [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021) - [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022) - [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024) - [OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems](https://mlsanity.com/paper/2402.14008/) (2024) - [Graph of Thoughts: Solving Elaborate Problems with Large Language Models](https://mlsanity.com/paper/2308.09687/) (2023) - [A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT](https://mlsanity.com/paper/2302.11382/) (2023) - [From System 1 to System 2: A Survey of Reasoning Large Language Models](https://mlsanity.com/paper/2502.17419/) (2025) - [SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models](https://mlsanity.com/paper/2307.10635/) (2023) - [Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4](https://mlsanity.com/paper/2304.03439/) (2023) - [Talking About Large Language Models](https://mlsanity.com/paper/2212.03551/) (2022) - [The Debate Over Understanding in AI's Large Language Models](https://mlsanity.com/paper/2210.13966/) (2022) - [Task Contamination: Language Models May Not Be Few-Shot Anymore](https://mlsanity.com/paper/2312.16337/) (2023) - [Language Model Behavior: A Comprehensive Survey](https://mlsanity.com/paper/2303.11504/) (2023) - [Beware of Metacognitive Laziness: Effects of Generative Artificial Intelligence on Learning Motivation, Processes, and Performance](https://mlsanity.com/paper/2412.09315/) (2024) - [Generative Pre-trained Transformer: A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directions](https://mlsanity.com/paper/2305.10435/) (2023) - [Empowering LLMs with Logical Reasoning: A Comprehensive Survey](https://mlsanity.com/paper/2502.15652/) (2025) - [Physics: Benchmarking Foundation Models on University-Level Physics Problem Solving](https://mlsanity.com/paper/2503.21821/) (2025) - [Can LLMs Master Math? Investigating Large Language Models on Math Stack Exchange](https://mlsanity.com/paper/2404.00344/) (2024) - [When a language model is optimized for reasoning, does it still show embers of autoregression? An analysis of OpenAI o1](https://mlsanity.com/paper/2410.01792/) (2024) - [AI and the FCI: Can ChatGPT Project an Understanding of Introductory Physics?](https://mlsanity.com/paper/2303.01067/) (2023) - [Examining the Potential and Pitfalls of ChatGPT in Science and Engineering Problem-Solving](https://mlsanity.com/paper/2310.08773/) (2023) - [Multilingual Performance of a Multimodal Artificial Intelligence System on Multisubject Physics Concept Inventories](https://mlsanity.com/paper/2501.06143/) (2025) - [Difficult Task Yes but Simple Task No: Unveiling the Laziness in Multimodal LLMs](https://mlsanity.com/paper/2410.11437/) (2024) ## Built on by (most cited first, 7 of 7) - [SciAgent: A Unified Multi-Agent System for Generalistic Scientific Reasoning](https://mlsanity.com/paper/2511.08151/) (2025) - [Evaluating NLP Embedding Models for Handling Science-Specific Symbolic Expressions in Student Texts](https://mlsanity.com/paper/2505.17950/) (2025) - [Daily and Weekly Periodicity in Large Language Model Performance and Its Implications for Research](https://mlsanity.com/paper/2602.15889/) (2026) - [Superstudent intelligence in thermodynamics](https://mlsanity.com/paper/2506.09822/) (2025) - [Emergent World Beliefs: Exploring Transformers in Stochastic Games](https://mlsanity.com/paper/2512.23722/) (2025) - [LLM-as-a-judge validity is strongly task-dependent across physics assessment formats](https://mlsanity.com/paper/2603.14732/) (2026) - [Assessing AI in Introductory Physics Problem Solving](https://mlsanity.com/paper/2607.14303/) (2026) ## Abstract (arXiv) Large language models (LLMs) are now widely accessible, reaching learners at all educational levels. This development has raised concerns that their use may circumvent essential learning processes and compromise the integrity of established assessment formats. In physics education, where problem solving plays a central role in instruction and assessment, it is therefore essential to understand the physics-specific problem-solving capabilities of LLMs. Such understanding is key to informing responsible and pedagogically sound approaches to integrating LLMs into instruction and assessment. This study therefore compares the problem-solving performance of a general-purpose LLM (GPT-4o, using varying prompting techniques) and a reasoning-optimized model (o1-preview) with that of participants of the German Physics Olympiad, based on a set of well-defined Olympiad problems. In addition to evaluating the correctness of the generated solutions, the study analyzes characteristic strengths and limitations of LLM-generated solutions. The findings of this study indicate that both tested LLMs (GPT-4o and o1-preview) demonstrate advanced problem-solving capabilities on Olympiad-type physics problems, on average outperforming the human participants. Prompting techniques had little effect on GPT-4o's performance, while o1-preview almost consistently outperformed both GPT-4o and the human benchmark. Based on these findings, the study discusses implications for the design of summative and formative assessment in physics education, including how to uphold assessment integrity and support students in critically engaging with LLMs.