# System 2 thinking in OpenAI's o1-preview model: Near-perfect performance on a mathematics exam arXiv: https://arxiv.org/abs/2410.07114 ยท HTML: https://mlsanity.com/paper/2410.07114/ ## Key facts - Cited by 10 papers in the map (data through 14 Oct 2026). - Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. - Builds on 21 of its 53 references that are in the map. - First posted on arXiv 19 Sep 2024. ## Placed in - LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/) ## Builds on (most cited first, 21 of 21) - [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023) - [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020) - [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022) - [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023) - [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022) - [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023) - [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023) - [Solving math word problems with process- and outcome-based feedback](https://mlsanity.com/paper/2211.14275/) (2022) - [Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspective](https://mlsanity.com/paper/2305.15408/) (2023) - [Mathematical Capabilities of ChatGPT](https://mlsanity.com/paper/2301.13867/) (2023) - [Chain of Thought Empowers Transformers to Solve Inherently Serial Problems](https://mlsanity.com/paper/2402.12875/) (2024) - [Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews](https://mlsanity.com/paper/2403.07183/) (2024) - [Benchmark Data Contamination of Large Language Models: A Survey](https://mlsanity.com/paper/2406.04244/) (2024) - [Delving into LLM-assisted writing in biomedical publications through excess vocabulary](https://mlsanity.com/paper/2406.07016/) (2024) - [The AI Review Lottery: Widespread AI-Assisted Peer Reviews Boost Paper Scores and Acceptance Rates](https://mlsanity.com/paper/2405.02150/) (2024) - [Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence](https://mlsanity.com/paper/2402.09880/) (2024) - [A Survey on Evaluation of Multimodal Large Language Models](https://mlsanity.com/paper/2408.15769/) (2024) - [Prompting Techniques for Reducing Social Bias in LLMs through System 1 and System 2 Cognitive Processes](https://mlsanity.com/paper/2404.17218/) (2024) - [ChatGPT "contamination": estimating the prevalence of LLMs in the scholarly literature](https://mlsanity.com/paper/2403.16887/) (2024) - [GPT-4 vs. Human Translators: A Comprehensive Evaluation of Translation Quality Across Languages, Domains, and Expertise Levels](https://mlsanity.com/paper/2407.03658/) (2024) - [Tangram: Benchmark for Evaluating Geometric Element Recognition in Large Multimodal Models](https://mlsanity.com/paper/2408.13854/) (2024) ## Built on by (most cited first, 10 of 10) - [Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Models](https://mlsanity.com/paper/2505.04921/) (2025) - [ARES: Multimodal Adaptive Reasoning via Difficulty-Aware Token-Level Entropy Shaping](https://mlsanity.com/paper/2510.08457/) (2025) - [ComfyUI-R1: Exploring Reasoning Models for Workflow Generation](https://mlsanity.com/paper/2506.09790/) (2025) - [CoAT: Chain-of-Associated-Thoughts Framework for Enhancing Large Language Models Reasoning](https://mlsanity.com/paper/2502.02390/) (2025) - [BELLE: A Bi-Level Multi-Agent Reasoning Framework for Multi-Hop Question Answering](https://mlsanity.com/paper/2505.11811/) (2025) - [SymbOmni: Evolving Agentic Omni Models via Symbolic Concept Learning](https://mlsanity.com/paper/2607.12042/) (2026) - [Overcoming State Inertia: Minimally Invasive Temporal Alignment for Evolving Contexts](https://mlsanity.com/paper/2512.03704/) (2025) - [The role of System 1 and System 2 semantic memory structure in human and LLM biases](https://mlsanity.com/paper/2604.12816/) (2026) - [Decision Protocols in Multi-Agent Large Language Model Conversations](https://mlsanity.com/paper/2607.05477/) (2026) - [From Mastery Profile to Simulated Response: Stochastic Student Knowledge Graphs (SSKG) for Faithful LLM Student Simulation](https://mlsanity.com/paper/2608.21668/) (2026) ## Abstract (arXiv) The processes underlying human cognition are often divided into System 1, which involves fast, intuitive thinking, and System 2, which involves slow, deliberate reasoning. Previously, large language models were criticized for lacking the deeper, more analytical capabilities of System 2. In September 2024, OpenAI introduced the o1 model series, designed to handle System 2-like reasoning. While OpenAI's benchmarks are promising, independent validation is still needed. In this study, we tested the o1-preview model twice on the Dutch 'Mathematics B' final exam. It scored a near-perfect 76 and 74 out of 76 points. For context, only 24 out of 16,414 students in the Netherlands achieved a perfect score. By comparison, the GPT-4o model scored 66 and 62 out of 76, well above the Dutch students' average of 40.63 points. Neither model had access to the exam figures. Since there was a risk of model contami-nation (i.e., the knowledge cutoff for o1-preview and GPT-4o was after the exam was published online), we repeated the procedure with a new Mathematics B exam that was published after the cutoff date. The results again indicated that o1-preview performed strongly (97.8th percentile), which suggests that contamination was not a factor. We also show that there is some variability in the output of o1-preview, which means that sometimes there is 'luck' (the answer is correct) or 'bad luck' (the output has diverged into something that is incorrect). We demonstrate that the self-consistency approach, where repeated prompts are given and the most common answer is selected, is a useful strategy for identifying the correct answer. It is concluded that while OpenAI's new model series holds great potential, certain risks must be considered.