# Are Language Models Puzzle Prodigies? Algorithmic Puzzles Unveil Serious Challenges in Multimodal Reasoning

arXiv: https://arxiv.org/abs/2403.03864 · HTML: https://mlsanity.com/paper/2403.03864/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 1 of its 24 references that are in the map.
- First posted on arXiv 6 Mar 2024.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)

## Builds on (most cited first, 1 of 1)

- [Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks](https://mlsanity.com/paper/2103.14749/) (2021)

## Built on by (most cited first, 25 of 25)

- [OctoTools: An Agentic Framework with Extensible Tools for Complex Reasoning](https://mlsanity.com/paper/2502.11271/) (2025)
- [Think or Not Think: A Study of Explicit Thinking in Rule-Based Visual Reinforcement Fine-Tuning](https://mlsanity.com/paper/2503.16188/) (2025)
- [OpenMMReasoner: Pushing the Frontiers for Multimodal Reasoning with an Open and General Recipe](https://mlsanity.com/paper/2511.16334/) (2025)
- [Open Vision Reasoner: Transferring Linguistic Cognitive Behavior for Visual Reasoning](https://mlsanity.com/paper/2507.05255/) (2025)
- [One RL to See Them All: Visual Triple Unified Reinforcement Learning](https://mlsanity.com/paper/2505.18129/) (2025)
- [From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Models](https://mlsanity.com/paper/2509.25373/) (2025)
- [InternBootcamp: Boosting LLM Reasoning with Verifiable Task Scaling](https://mlsanity.com/paper/2508.08636/) (2025)
- [MIRAGE: Assessing Hallucination in Multimodal Reasoning Chains of MLLM](https://mlsanity.com/paper/2505.24238/) (2025)
- [Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?](https://mlsanity.com/paper/2501.02669/) (2025)
- [Sponge Tool Attack: Stealthy Denial-of-Efficiency against Tool-Augmented Agentic Reasoning](https://mlsanity.com/paper/2601.17566/) (2026)
- [Lost in Time: Clock and Calendar Understanding Challenges in Multimodal LLMs](https://mlsanity.com/paper/2502.05092/) (2025)
- [PuzzleWorld: A Benchmark for Multimodal, Open-Ended Reasoning in Puzzlehunts](https://mlsanity.com/paper/2506.06211/) (2025)
- [Seeing Is Believing? A Benchmark for Multimodal Large Language Models on Visual Illusions and Anomalies](https://mlsanity.com/paper/2602.01816/) (2026)
- [SIRI-Bench: Challenging VLMs' Spatial Intelligence through Complex Reasoning Tasks](https://mlsanity.com/paper/2506.14512/) (2025)
- [What MLLMs Learn about When they Learn about Multimodal Reasoning](https://mlsanity.com/paper/2510.01719/) (2025)
- [ScratchEval: Are GPT-4o Smarter than My Child? Evaluating Large Multimodal Models with Visual Programming Challenges](https://mlsanity.com/paper/2411.18932/) (2024)
- [The Jumping Reasoning Curve? Tracking the Evolution of Reasoning Performance in GPT-[n] and o-[n] Models on Multimodal Puzzles](https://mlsanity.com/paper/2502.01081/) (2025)
- [Look Light, Think Heavy: What Multimodal Chain-of-Thought Reasoning Can and Cannot Do](https://mlsanity.com/paper/2606.22565/) (2026)
- [Abductive Symbolic Solver on Abstraction and Reasoning Corpus](https://mlsanity.com/paper/2411.18158/) (2024)
- [DataComp-VLM: Improved Open Datasets for Vision-Language Models](https://mlsanity.com/paper/2606.28551/) (2026)
- [From Perception to Reasoning: Deep Thinking Empowers Multimodal Large Language Models](https://mlsanity.com/paper/2511.12861/) (2025)
- [Imagine in Space: Exploring the Frontier of Spatial Intelligence and Reasoning Efficiency in Vision Language Models](https://mlsanity.com/paper/2511.13782/) (2025)
- [AnE: Pushing the Reasoning Frontier of Multimodal LLMs via Anchor Evolution](https://mlsanity.com/paper/2605.25571/) (2026)
- [Look Where You Say You're Looking: Self-Grounded Attention for Visual Reasoning](https://mlsanity.com/paper/2610.05023/) (2026)
- [CREATE: Creativity Evaluation of VLMs using the Visual Remote Associates Test](https://mlsanity.com/paper/2610.18569/) (2026)

## Abstract (arXiv)

This paper introduces the novel task of multimodal puzzle solving, framed within the context of visual question-answering. We present a new dataset, AlgoPuzzleVQA designed to challenge and evaluate the capabilities of multimodal language models in solving algorithmic puzzles that necessitate both visual understanding, language understanding, and complex algorithmic reasoning. We create the puzzles to encompass a diverse array of mathematical and algorithmic topics such as boolean logic, combinatorics, graph theory, optimization, search, etc., aiming to evaluate the gap between visual data interpretation and algorithmic problem-solving skills. The dataset is generated automatically from code authored by humans. All our puzzles have exact solutions that can be found from the algorithm without tedious human calculations. It ensures that our dataset can be scaled up arbitrarily in terms of reasoning complexity and dataset size. Our investigation reveals that large language models (LLMs) such as GPT4V and Gemini exhibit limited performance in puzzle-solving tasks. We find that their performance is near random in a multi-choice question-answering setup for a significant number of puzzles. The findings emphasize the challenges of integrating visual, language, and algorithmic knowledge for solving complex reasoning problems.