From Pixels to BFS: High Maze Accuracy Does Not Imply Visual Planning
Not on the map. Why?
Abstract (arXiv)
How do multimodal models solve visual spatial tasks -- through genuine planning, or through brute-force search in token space? We introduce textscMazeBench, a benchmark of 110 procedurally generated maze images across nine controlled groups, and evaluate 16 model configurations from OpenAI, Anthropic, Google, and Alibaba. GPT-5.4 solves 91% and Gemini 3.1 Pro 79%, but these scores are misleading: models typically translate images into text grids and then enumerate paths step by step, consuming 1,710--22,818 tokens per solve for a task humans do quickly. Without added reasoning budgets, all configurations score only 2--12%; on 20×20 ultra-hard mazes, they hit token limits and fail. Qualitative traces reveal a common two-stage strategy: image-to-grid translation followed by token-level search, effectively BFS in prose. A text-grid ablation shows Claude Sonnet 4.6 rising from 6% on images to 80% when given the correct grid, isolating weak visual extraction from downstream search. When explicitly instructed not to construct a grid or perform graph search, models still revert to the same enumeration strategy. textscMazeBench therefore shows that high accuracy on visual planning tasks does not imply human-like spatial understanding.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 12 of its 19 references that are in the map. First posted on arXiv 27 Mar 2026.
Builds on
12 of 19 references.
References · 12
- 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
- 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
- 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
- 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
- 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
- 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
- 2025Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmarkcited by 132
- 2025Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokenscited by 65
- 2025SpatialBench: Benchmarking Multimodal Large Language Models for Spatial Cognitioncited by 35
- 2024GSR-BENCH: A Benchmark for Grounded Spatial Reasoning Evaluation via Multimodal LLMscited by 33
- 2024MMEvalPro: Calibrating Multimodal Benchmarks Towards Trustworthy and Efficient Evaluationcited by 17
- 2026SpatiaLab: Can Vision-Language Models Perform Spatial Reasoning in the Wild?cited by 10
Built on by
Citing papers per year: 2026: 1.
1 paper.