mlsanityData to 7 Oct 2026

A Systematic Assessment of OpenAI o1-Preview for Higher Order Thinking in Education

arXiv 2410.21287 ↗First posted 11 Oct 2024cs.CY cs.AIcited by 16

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

As artificial intelligence (AI) continues to advance, it demonstrates capabilities comparable to human intelligence, with significant potential to transform education and workforce development. This study evaluates OpenAI o1-preview's ability to perform higher-order cognitive tasks across 14 dimensions, including critical thinking, systems thinking, computational thinking, design thinking, metacognition, data literacy, creative thinking, abstract reasoning, quantitative reasoning, logical reasoning, analogical reasoning, and scientific reasoning. We used validated instruments like the Ennis-Weir Critical Thinking Essay Test and the Biological Systems Thinking Test to compare the o1-preview's performance with human performance systematically. Our findings reveal that o1-preview outperforms humans in most categories, achieving 150% better results in systems thinking, computational thinking, data literacy, creative thinking, scientific reasoning, and abstract reasoning. However, compared to humans, it underperforms by around 25% in logical reasoning, critical thinking, and quantitative reasoning. In analogical reasoning, both o1-preview and humans achieved perfect scores. Despite these strengths, the o1-preview shows limitations in abstract reasoning, where human psychology students outperform it, highlighting the continued importance of human oversight in tasks requiring high-level abstraction. These results have significant educational implications, suggesting a shift toward developing human skills that complement AI, such as creativity, abstract reasoning, and critical thinking. This study emphasizes the transformative potential of AI in education and calls for a recalibration of educational goals, teaching methods, and curricula to align with an AI-driven world.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 48 of its 186 references that are in the map. First posted on arXiv 11 Oct 2024.

Builds on

48 of 186 references.

References · 48

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  4. 2023Let's Verify Step by Stepcited by 3,598
  5. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
  6. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
  7. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 752
  8. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 350
  9. 2023Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology Viewcited by 221
  10. 2024Chain of Agents: Large Language Models Collaborating on Long-Context Taskscited by 201
  11. 2023RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Modelscited by 173
  12. 2024Evaluation of OpenAI o1: Opportunities and Challenges of AGIcited by 127
36 more
  1. 2024Simulating Classroom Education with LLM-Empowered Agentscited by 125
  2. 2023Solving Math Word Problems by Combining Language Models With Symbolic Solverscited by 124
  3. 2023Learning Performance-Improving Code Editscited by 103
  4. 2024LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Modelscited by 97
  5. 2024LLMs Still Can't Plan; Can LRMs? A Preliminary Evaluation of OpenAI's o1 on PlanBenchcited by 93
  6. 2024How to Train Data-Efficient LLMscited by 91
  7. 2024SOTOPIA-π: Interactive Learning of Socially Intelligent Language Agentscited by 89
  8. 2024Metacognitive Capabilities of LLMs: An Exploration in Mathematical Problem Solvingcited by 82
  9. 2023LLMs and the Abstraction and Reasoning Corpus: Successes, Failures, and the Importance of Object-based Representationscited by 81
  10. 2024Scaling Instructable Agents Across Many Simulated Worldscited by 67
  11. 2024Are LLMs Capable of Data-based Statistical and Causal Reasoning? Benchmarking Advanced Quantitative Reasoning with Datacited by 61
  12. 2024Learning to Decode Collaboratively with Multiple Language Modelscited by 61
  13. 2024Exploring Knowledge Tracing in Tutor-Student Dialogues using LLMscited by 44
  14. 2024Using Counterfactual Tasks to Evaluate the Generality of Analogical Reasoning in Large Language Modelscited by 38
  15. 2024Self-Reflection in LLM Agents: Effects on Problem-Solving Performancecited by 38
  16. 2023In-Context Analogical Reasoning with Pre-Trained Language Modelscited by 22
  17. 2024How AI Ideas Affect the Creativity, Diversity, and Evolution of Human Ideas: Evidence From a Large, Dynamic Experimentcited by 22
  18. 2024From Calculation to Adjudication: Examining LLM judges on Mathematical Reasoning Taskscited by 22
  19. 2023Multimodality of AI for Education: Towards Artificial General Intelligencecited by 21
  20. 2024Advancing Social Intelligence in AI Agents: Technical Challenges and Open Questionscited by 21
  21. 2024LLMs as Models for Analogical Reasoningcited by 16
  22. 2024Assessing Large Language Models in Mechanical Engineering Education: A Study on Mechanics-Focused Conceptual Understandingcited by 15
  23. 2024From MOOC to MAIC: Reshaping Online Teaching and Learning through LLM-driven Agentscited by 15
  24. 2023NERIF: GPT-4V for Automatic Scoring of Drawn Modelscited by 13
  25. 2024SPL: A Socratic Playground for Learning Powered by Large Language Modelcited by 9
  26. 2024Relevant or Random: Can LLMs Truly Perform Analogical Reasoning?cited by 8
  27. 2024Transforming Teachers' Roles and Agencies in the Era of Generative AI: Perceptions, Acceptance, Knowledge, and Practicescited by 8
  28. 2023AGI: Artificial General Intelligence for Educationcited by 7
  29. 2024Can GPT-O1 Kill All Bugs? An Evaluation of GPT-Family LLMs on QuixBugscited by 7
  30. 2024Knowledge Tagging with Large Language Model based Multi-Agent Systemcited by 5
  31. 2024Industry 6.0: New Generation of Industry driven by Generative AI and Swarm of Heterogeneous Robotscited by 5
  32. 2024Generative AI as a metacognitive agent: A comparative mixed-method study with human participants on ICF-mimicking exam performancecited by 4
  33. 2024Enhancing LLM Problem Solving with REAP: Reflection, Explicit Problem Deconstruction, and Advanced Promptingcited by 4
  34. 2023Automatic Scoring of Students' Science Writing Using Hybrid Neural Networkcited by 3
  35. 2024Fast Analysis of the OpenAI O1-Preview Model in Solving Random K-SAT Problem: Does the LLM Solve the Problem Itself or Call an External SAT Solver?cited by 3
  36. 2024When ChatGPT is gone: Creativity reverts and homogeneity persistscited by 2

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 13.

16 papers.

Sort:

By area:

  1. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 291
  2. 2025InftyThink: Breaking the Length Limits of Long-Context Reasoning in Large Language Modelscited by 53
  3. 2025UAVs Meet LLMs: Overviews and Perspectives Toward Agentic Low-Altitude Mobilitycited by 48
  4. 2024Transcending Language Boundaries: Harnessing LLMs for Low-Resource Language Translationcited by 12
  5. 2024Towards Learning to Reason: Comparing LLMs with Neuro-Symbolic on Arithmetic Relations in Abstract Reasoningcited by 7
  6. 2025Towards Adaptive Feedback with AI: Comparing the Feedback Quality of LLMs and Teachers on Experimentation Protocolscited by 6
  7. 2024Foundation Models for Low-Resource Language Education (Vision Paper)cited by 5
  8. 2025Can Large Reasoning Models do Analogical Reasoning under Perceptual Uncertainty?cited by 5
  9. 2025Mathematical Computation and Reasoning Errors by Large Language Modelscited by 4
  10. 2025Large Language Models for Bioinformaticscited by 3
  11. 2025Exploring the Potential of Large Language Models for Estimating the Reading Comprehension Question Difficultycited by 3
  12. 2025THiNK: Can Large Language Models Think-aloud?cited by 3
4 more
  1. 2025Fact or Guesswork? Evaluating Large Language Models' Medical Knowledge with Structured One-Hop Judgmentscited by 2
  2. 2025DiCoRe: Enhancing Zero-shot Event Detection via Divergent-Convergent LLM Reasoningcited by 2
  3. 2025ADLGen: Synthesizing Symbolic, Event-Triggered Sensor Sequences for Human Activity Modelingcited by 0
  4. 2025Exploring Communication Strategies for Collaborative LLM Agents in Mathematical Problem-Solvingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Can OpenAI o1 outperform humans in higher-order cognitive thinking?
  2. 2024System 2 thinking in OpenAI's o1-preview model: Near-perfect performance on a mathematics exam
  3. 2025Designing AI Systems that Augment Human Performed vs. Demonstrated Critical Thinking
  4. 2024OpenAI-o1 AB Testing: Does the o1 model really do good reasoning in math problem solving?
  5. 2024O1 Replication Journey -- Part 2: Surpassing O1-preview through Simple Distillation, Big Progress or Bitter Lesson?
  6. 2024The Cognitive Capabilities of Generative AI: A Comparative Analysis with Human Benchmarks
  7. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs
  8. 2024Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worse