# The Jumping Reasoning Curve? Tracking the Evolution of Reasoning Performance in GPT-[n] and o-[n] Models on Multimodal Puzzles

arXiv: https://arxiv.org/abs/2502.01081 · HTML: https://mlsanity.com/paper/2502.01081/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 14 of its 19 references that are in the map.
- First posted on arXiv 3 Feb 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)

## Builds on (most cited first, 14 of 14)

- [On the Measure of Intelligence](https://mlsanity.com/paper/1911.01547/) (2019)
- [PuzzleVQA: Diagnosing Multimodal Reasoning Challenges of Language Models with Abstract Visual Patterns](https://mlsanity.com/paper/2403.13315/) (2024)
- [Puzzle Solving using Reasoning of Large Language Models: A Survey](https://mlsanity.com/paper/2402.11291/) (2024)
- [Computing Power and the Governance of Artificial Intelligence](https://mlsanity.com/paper/2402.08797/) (2024)
- [Are Language Models Puzzle Prodigies? Algorithmic Puzzles Unveil Serious Challenges in Multimodal Reasoning](https://mlsanity.com/paper/2403.03864/) (2024)
- [True Detective: A Deep Abductive Reasoning Benchmark Undoable for GPT-3 and Challenging for GPT-4](https://mlsanity.com/paper/2212.10114/) (2022)
- [Large language models for artificial general intelligence (AGI): A survey of foundational principles and approaches](https://mlsanity.com/paper/2501.03151/) (2025)
- [Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad?](https://mlsanity.com/paper/2410.19546/) (2024)
- [PUZZLES: A Benchmark for Neural Algorithmic Reasoning](https://mlsanity.com/paper/2407.00401/) (2024)
- [Unraveling the ARC Puzzle: Mimicking Human Solutions with Object-Centric Decision Transformer](https://mlsanity.com/paper/2306.08204/) (2023)
- [REBUS: A Robust Evaluation Benchmark of Understanding Symbols](https://mlsanity.com/paper/2401.05604/) (2024)
- [AGI: Artificial General Intelligence for Education](https://mlsanity.com/paper/2304.12479/) (2023)
- [Machine Psychology: Integrating Operant Conditioning with the Non-Axiomatic Reasoning System for Advancing Artificial General Intelligence Research](https://mlsanity.com/paper/2405.19498/) (2024)
- [What is Meant by AGI? On the Definition of Artificial General Intelligence](https://mlsanity.com/paper/2404.10731/) (2024)

## Built on by (most cited first, 4 of 4)

- [Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Models](https://mlsanity.com/paper/2504.21277/) (2025)
- [Disentangling Perception and Reasoning in Multimodal LLMs via Reward Design](https://mlsanity.com/paper/2601.00215/) (2026)
- [From Perception to Reasoning: Deep Thinking Empowers Multimodal Large Language Models](https://mlsanity.com/paper/2511.12861/) (2025)
- [PuzzleMate: Benchmarking MLLMs for Egocentric Puzzle Assistance](https://mlsanity.com/paper/2609.14473/) (2026)

## Abstract (arXiv)

The releases of OpenAI's o-[n] series, such as o1, o3, and o4-mini, mark a significant paradigm shift in Large Language Models towards advanced reasoning capabilities. Notably, models like o3 have demonstrated strong performance on benchmarks like the Abstraction and Reasoning Corpus for Artificial General Intelligence (ARC-AGI). However, this benchmark is limited to symbolic patterns, whereas humans often perceive and reason about multimodal scenarios involving both vision and language data. Thus, there is an urgent need to investigate advanced reasoning capabilities in multimodal tasks. To this end, we track the evolution of the GPT-[n] and o-[n] series models (including o1, o3, and o4-mini) on challenging multimodal puzzles from PuzzleVQA and AlgoPuzzleVQA, which demand fine-grained visual perception. Our results reveal that o-[n] series, particularly later iterations like o3 and o4-mini, significantly outperform the GPT-[n] series and show strong scalability in multimodal reasoning. Nonetheless, despite these substantial advancements and the superior capabilities demonstrated by the o-[n] series, our findings highlight that even these leading models face persistent challenges. Difficulties are particularly evident in tasks requiring precise visual perception, robust compositional reasoning across multiple visual attributes, and solving complex algorithmic or highly combinatorial puzzles, indicating critical areas for future AGI development. We plan to continuously track new models in the series and update our results in this paper accordingly. All resources used in this evaluation are openly available at https://github.com/declare-lab/LLM-PuzzleTest.