Data to 7 Oct 2026

Evaluation of OpenAI o1: Opportunities and Challenges of AGI

arXiv 2409.18486 ↗First posted 27 Sep 2024cs.CLcited by 127

Not on the map. Why?

Cited by 71 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This comprehensive study evaluates the performance of OpenAI's o1-preview large language model across a diverse array of complex reasoning tasks, spanning multiple domains, including computer science, mathematics, natural sciences, medicine, linguistics, and social sciences. Through rigorous testing, o1-preview demonstrated remarkable capabilities, often achieving human-level or superior performance in areas ranging from coding challenges to scientific reasoning and from language processing to creative problem-solving. Key findings include: -83.3% success rate in solving complex competitive programming problems, surpassing many human experts. -Superior ability in generating coherent and accurate radiology reports, outperforming other evaluated models. -100% accuracy in high school-level mathematical reasoning tasks, providing detailed step-by-step solutions. -Advanced natural language inference capabilities across general and specialized domains like medicine. -Impressive performance in chip design tasks, outperforming specialized models in areas such as EDA script generation and bug analysis. -Remarkable proficiency in anthropology and geology, demonstrating deep understanding and reasoning in these specialized fields. -Strong capabilities in quantitative investing. O1 has comprehensive financial knowledge and statistical modeling skills. -Effective performance in social media analysis, including sentiment analysis and emotion recognition. The model excelled particularly in tasks requiring intricate reasoning and knowledge integration across various fields. While some limitations were observed, including occasional errors on simpler problems and challenges with certain highly specialized concepts, the overall results indicate significant progress towards artificial general intelligence.

Key facts. Cited by 127 papers in the map (data through 7 Oct 2026). First posted on arXiv 27 Sep 2024.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 19; 2025: 97; 2026: 11.

127 papers.

Sort:

By area:

2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Stepcited by 446
2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 291
2024HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMscited by 257
2025The Danger of Overthinking: Examining the Reasoning-Action Dilemma in Agentic Taskscited by 150
2024Marco-o1: Towards Open Reasoning Models for Open-Ended Solutionscited by 142
2025MedReason: Eliciting Factual Medical Reasoning Steps in LLMs via Knowledge Graphscited by 105
2025DeepSeek-R1 Thoughtology: Let's think about LLM Reasoningcited by 91
2025From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discoverycited by 74
2024Game-theoretic LLM: Agent Workflow for Negotiation Gamescited by 55
2024A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challengescited by 55
2025InftyThink: Breaking the Length Limits of Long-Context Reasoning in Large Language Modelscited by 53

38 more

2025Exploring the Limit of Outcome Reward for Learning Mathematical Reasoningcited by 51
2025Inverse Scaling in Test-Time Computecited by 49
2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
2025Easier Painting Than Thinking: Can Text-to-Image Models Set the Stage, but Not Direct the Play?cited by 33
2025Fino1: On the Transferability of Reasoning-Enhanced LLMs and Reinforcement Learning to Financecited by 32
2025Generative to Agentic AI: Survey, Conceptualization, and Challengescited by 32
2025InSTA: Towards Internet-Scale Training For Agentscited by 31
2025Benchmarking Reasoning Robustness in Large Language Modelscited by 30
2024Large Language Models Can Self-Improve in Long-context Reasoningcited by 29
2025Quantifying the Reasoning Abilities of LLMs on Real-world Clinical Casescited by 28
2025ReviewAgents: Bridging the Gap Between Human and AI-Generated Paper Reviewscited by 28
2024DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compactioncited by 26
2024Large Language Models for Manufacturingcited by 24
2025MathAgent: Leveraging a Mixture-of-Math-Agent Framework for Real-World Multimodal Mathematical Error Detectioncited by 24
2025Stay Focused: Problem Drift in Multi-Agent Debatecited by 23
2025DianJin-R1: Evaluating and Enhancing Financial Reasoning in Large Language Modelscited by 22
2026Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokenscited by 21
2025AutoP2C: An LLM-Based Agent Framework for Code Repository Generation from Multimodal Content in Academic Paperscited by 18
2025A Survey of Slow Thinking-based Reasoning LLMs using Reinforced Learning and Inference-time Scaling Lawcited by 18
2025THINK-Bench: Evaluating Thinking Efficiency and Chain-of-Thought Quality of Large Reasoning Modelscited by 18
2025ThinkBench: Dynamic Out-of-Distribution Evaluation for Robust LLM Reasoningcited by 17
2025Disentangling Reasoning and Knowledge in Medical Large Language Modelscited by 17
2024A Systematic Assessment of OpenAI o1-Preview for Higher Order Thinking in Educationcited by 16
2025ThinkPatterns-21k: A Systematic Study on the Impact of Thinking Patterns in LLMscited by 16
2025RL of Thoughts: Navigating LLM Reasoning with Inference-time Reinforcement Learningcited by 16
2025Synthetic Data RL: Task Definition Is All You Needcited by 15
2025Marco-o1 v2: Towards Widening The Distillation Bottleneck for Reasoning Modelscited by 13
2025Joint Evaluation of Answer and Reasoning Consistency for Hallucination Detection in Large Reasoning Modelscited by 13
2024Transcending Language Boundaries: Harnessing LLMs for Low-Resource Language Translationcited by 12
2025Reasoning-as-Logic-Units: Scaling Test-Time Reasoning in Large Language Models Through Logic Unit Alignmentcited by 12
2025HiBench: Benchmarking LLMs Capability on Hierarchical Structure Reasoningcited by 12
2025Are Language Models Consequentialist or Deontological Moral Reasoners?cited by 12
2025Transformers Provably Learn Chain-of-Thought Reasoning with Length Generalizationcited by 12
2024Towards Next-Generation Medical Agent: How o1 is Reshaping Decision-Making in Medical Scenarioscited by 10
2024Rethinking Chain-of-Thought from the Perspective of Self-Trainingcited by 10
2024ShapefileGPT: A Multi-Agent Large Language Model Framework for Automated Shapefile Processingcited by 9
2025CMCTS: A Constrained Monte Carlo Tree Search Framework for Mathematical Reasoning in Large Language Modelcited by 9
2025Aurelia: Test-time Reasoning Distillation in Audio-Visual LLMscited by 9

50 most cited of 127 All 127 (JSON)

Similar, not cited

Close in wording, no citation either way