# Understanding and Benchmarking Artificial Intelligence: OpenAI's o3 Is Not AGI

arXiv: https://arxiv.org/abs/2501.07458 · HTML: https://mlsanity.com/paper/2501.07458/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Jan 2025.

## Built on by (most cited first, 19 of 19)

- [Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models](https://mlsanity.com/paper/2503.09567/) (2025)
- [Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Models](https://mlsanity.com/paper/2505.04921/) (2025)
- [Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluation](https://mlsanity.com/paper/2502.06559/) (2025)
- [Table-R1: Inference-Time Scaling for Table Reasoning](https://mlsanity.com/paper/2505.23621/) (2025)
- [LoopViT: Scaling Visual ARC with Looped Transformers](https://mlsanity.com/paper/2602.02156/) (2026)
- [GeoGramBench: Benchmarking the Geometric Program Reasoning in Modern LLMs](https://mlsanity.com/paper/2505.17653/) (2025)
- [HardML: A Benchmark For Evaluating Data Science And Machine Learning knowledge and reasoning in AI](https://mlsanity.com/paper/2501.15627/) (2025)
- [GIFARC: Synthetic Dataset for Leveraging Human-Intuitive Analogies to Elevate AI Reasoning](https://mlsanity.com/paper/2505.20672/) (2025)
- [Code-driven Number Sequence Calculation: Enhancing the inductive Reasoning Abilities of Large Language Models](https://mlsanity.com/paper/2510.14620/) (2025)
- [A Representationalist, Functionalist and Naturalistic Conception of Intelligence as a Foundation for AGI](https://mlsanity.com/paper/2503.07600/) (2025)
- [Semantic Deception: When Reasoning Models Can't Compute an Addition](https://mlsanity.com/paper/2512.20812/) (2025)
- [A Conceptual Framework for AI Capability Evaluations](https://mlsanity.com/paper/2506.18213/) (2025)
- [Improving AGI Evaluation: A Data Science Perspective](https://mlsanity.com/paper/2510.01687/) (2025)
- [On the Measure of a Model: From Intelligence to Generality](https://mlsanity.com/paper/2511.11773/) (2025)
- [PromptTailor: Multi-turn Intent-Aligned Prompt Synthesis for Lightweight LLMs](https://mlsanity.com/paper/2511.21725/) (2025)
- [WISE: Weighted Iterative Society-of-Experts for Robust Multimodal Multi-Agent Debate](https://mlsanity.com/paper/2512.02405/) (2025)
- [Strategies for Creating Uncertainty in the AI Era to Trigger Students Critical Thinking: Pedagogical Design, Assessment Rubric, and Exam System](https://mlsanity.com/paper/2602.00026/) (2026)
- [On the Exact Algorithmic Extraction of Finite Tesselations Through Prime Extraction of Minimal Representative Forms](https://mlsanity.com/paper/2603.00911/) (2026)
- [Rethinking Stepwise Model Routing: A Cost-Efficient Table Reasoning Perspective](https://mlsanity.com/paper/2605.29319/) (2026)

## Abstract (arXiv)

OpenAI's o3 achieves a high score of 87.5 % on ARC-AGI, a benchmark proposed to measure intelligence. This raises the question whether systems based on Large Language Models (LLMs), particularly o3, demonstrate intelligence and progress towards artificial general intelligence (AGI). Building on the distinction between skills and intelligence made by Franccois Chollet, the creator of ARC-AGI, a new understanding of intelligence is introduced: an agent is the more intelligent, the more efficiently it can achieve the more diverse goals in the more diverse worlds with the less knowledge. An analysis of the ARC-AGI benchmark shows that its tasks represent a very specific type of problem that can be solved by massive trialling of combinations of predefined operations. This method is also applied by o3, achieving its high score through the extensive use of computing power. However, for most problems in the physical world and in the human domain, solutions cannot be tested in advance and predefined operations are not available. Consequently, massive trialling of predefined operations, as o3 does, cannot be a basis for AGI - instead, new approaches are required that can reliably solve a wide variety of problems without existing skills. To support this development, a new benchmark for intelligence is outlined that covers a much higher diversity of unknown tasks to be solved, thus enabling a comprehensive assessment of intelligence and of progress towards AGI.