mlsanityData to 14 Oct 2026

LEGO-Puzzles: How Good Are MLLMs at Multi-Step Spatial Reasoning?

arXiv 2503.19990 ↗First posted 25 Mar 2025cs.AIcited by 49

On the map: LLM reasoning → Spatial and embodied reasoning

Abstract (arXiv)

Many real-world applications of spatial intelligence, such as robotic control, autonomous driving, and automated assembly, require spatial reasoning across multiple sequential steps. However, the extent to which current Multimodal Large Language Models (MLLMs) possess this capability remains largely unexplored. Inspired by LEGO construction, a recreational activity that critically relies on multi-step spatial reasoning, we introduce LEGO-Puzzles: a benchmark designed to systematically evaluate the spatial reasoning capabilities of MLLMs from basic spatial understanding to multi-step planning. LEGO-Puzzles contains two task sets. The Elementary set covers 11 visual question-answering (VQA) tasks with 1,100 carefully curated samples to test elementary spatial reasoning skills that are cruical for LEGO assembly. The Planning set directly requires the model to generate a step-by-step plan for assembling a target LEGO structure, where the tasks are organized into subsets with different planning horizons ranging up to 8. Our evaluation of 29 state-of-the-art MLLMs shows that even the strongest models struggle with elementary reasoning tasks in LEGO construction, falling at least 20% behind human performance. The planning accuracy also quickly drops to 0% as the number of planning steps increases, whereas our human participants solve all the tasks perfectly. Switching the output format from multiple choice to image generation degrades model performance even further, leading to zero accuracy even for planning 3 steps. Overall, LEGO-Puzzles reveals critical limitations in current MLLMs' spatial reasoning capabilities and highlights the need for substantial advances.

Key facts. Cited by 49 papers in the map (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 48 of its 80 references that are in the map. First posted on arXiv 25 Mar 2025.

Builds on

48 of 80 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024CAD-GPT: Synthesising CAD Construction Sequence with Spatial Reasoning-Enhanced Multimodal LLMs27 Dec
  2. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces18 Dec
  3. 20243DSRBench: A Comprehensive 3D Spatial Reasoning Benchmark10 Dec
  4. 2024SURDS: Benchmarking Spatial Understanding and Reasoning in Driving Scenarios with Vision Language Models20 Nov
  5. 2024An Empirical Analysis on Spatial Reasoning Capabilities of Large Multimodal Models9 Nov
  6. 2024ActPlan-1K: Benchmarking the Procedural Planning Ability of Visual Language Models in Household Activities4 Oct
  7. 2024Reasoning Paths with Reference Objects Elicit Quantitative Spatial Reasoning in Large Vision-Language Models15 Sep
  8. 2024Coarse Correspondences Boost Spatial-Temporal Reasoning in Multimodal Language Model1 Aug
  9. 2023ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation24 Dec

Other references · 39

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2025Qwen2.5-VL Technical Reportcited by 5,587
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  7. 2023Qwen Technical Reportcited by 3,344
  8. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  9. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  10. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  11. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  12. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
27 more
  1. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  2. 2019Towards VQA Models That Can Readcited by 1,718
  3. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  4. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  5. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  6. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  7. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  8. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  9. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  10. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  11. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 572
  12. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
  13. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  14. 2023Generating Images with Multimodal Language Modelscited by 260
  15. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  16. 2025Imagine while Reasoning in Space: Multimodal Visualization-of-Thoughtcited by 206
  17. 2024Pixtral 12Bcited by 179
  18. 2024Building and better understanding vision-language models: insights and future directionscited by 173
  19. 2024ANOLE: An Open, Autoregressive, Native Large Multimodal Models for Interleaved Image-Text Generationcited by 114
  20. 2024SAT: Dynamic Spatial Aptitude Training for Multimodal Language Modelscited by 114
  21. 2022Super-CLEVR: A Virtual Benchmark to Diagnose Domain Robustness in Visual Reasoningcited by 92
  22. 2024MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Modelscited by 33
  23. 2024Holistic Evaluation for Interleaved Text-and-Image Generationcited by 28
  24. 2025Redundancy Principles for MLLMs Benchmarkscited by 19
  25. 20233D-Aware Visual Question Answering about Parts, Poses and Occlusionscited by 16
  26. 2025Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?cited by 14
  27. 2025FaceShot: Bring Any Character into Lifecited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 26; 2026: 23.

49 papers.

Sort:

By area:

  1. 2025MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligencecited by 183
  2. 2025Envisioning Beyond the Pixels: Benchmarking Reasoning-Informed Visual Editingcited by 99
  3. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 43
  4. 2025Beyond Seeing: Evaluating Multimodal LLMs on Tool-Enabled Image Perception, Transformation, and Reasoningcited by 29
  5. 2025STAR-R1: Spatial TrAnsformation Reasoning by Reinforcing Multimodal LLMscited by 21
  6. 2025SpaceVista: All-Scale Visual Spatial Reasoning from mm to kmcited by 19
  7. 2025SpatialThinker: Reinforcing Scene Graph-Grounded Spatial Reasoning via Dense Rewardscited by 16
  8. 2025GeoGramBench: Benchmarking the Geometric Program Reasoning in Modern LLMscited by 13
  9. 2025ViC-Bench: Benchmarking Visual-Interleaved Chain-of-Thought Capability in MLLMs with Free-Style Intermediate State Representationscited by 12
  10. 2025Actial: Activate Spatial Reasoning Ability of Multimodal Large Language Modelscited by 10
  11. 2025Jigsaw-Puzzles: From Seeing to Understanding to Reasoning in Vision-Language Modelscited by 9
  12. 2025CharacterShot: Controllable and Consistent 4D Character Animationcited by 9
37 more
  1. 2025A Lightweight Large Language Model-Based Multi-Agent System for 2D Frame Structural Analysiscited by 9
  2. 2025Spatial-DISE: A Unified Benchmark for Evaluating Spatial Reasoning in Vision-Language Modelscited by 9
  3. 2025CamPVG: Camera-Controlled Panoramic Video Generation with Epipolar-Aware Diffusioncited by 8
  4. 2025ORIGAMISPACE: Benchmarking Multimodal LLMs in Multi-Step Spatial Reasoning with Mathematical Constraintscited by 7
  5. 2026Can LLMs See Without Pixels? Benchmarking Spatial Intelligence from Textual Descriptionscited by 7
  6. 2026What's Holding Back Latent Visual Reasoning?cited by 7
  7. 2026Do MLLMs Really Understand Space? A Mathematical Reasoning Evaluationcited by 6
  8. 2025Hyperphantasia: A Benchmark for Evaluating the Mental Visualization Capabilities of Multimodal LLMscited by 5
  9. 2026Vesta: A Generalist Embodied Reasoning Modelcited by 5
  10. 2025Diagnosing Corruption-Induced Reliability Failures in Vision-Language Modelscited by 4
  11. 2025GamiBench: Evaluating Spatial Reasoning and 2D-to-3D Planning Capabilities of MLLMs with Origami Folding Taskscited by 4
  12. 2026TangramPuzzle: Evaluating Multimodal Large Language Models with Compositional Spatial Reasoningcited by 4
  13. 2025MineAnyBuild: Benchmarking Spatial Planning for Open-world AI Agentscited by 3
  14. 2025Unleashing Perception-Time Scaling to Multimodal Reasoning Modelscited by 3
  15. 2025GRAID: Enhancing Spatial Reasoning of VLMs Through High-Fidelity Data Generationcited by 3
  16. 2025Spatial4D-Bench: A Versatile 4D Spatial Intelligence Benchmarkcited by 3
  17. 2026Reasoning or Pattern Matching? Probing Large Vision-Language Models with Visual Puzzlescited by 2
  18. 2026TPRU: Advancing Temporal and Procedural Understanding in Large Multimodal Modelscited by 2
  19. 2026How Far Are Vision-Language Models from Constructing the Real World? A Benchmark for Physical Generative Reasoningcited by 2
  20. 2026SATURN: Symbolic Spatial Reasoning for Multi-Perspective Groundingcited by 2
  21. 2025From Macro to Micro: Benchmarking Microscopic Spatial Intelligence on Molecules via Vision-Language Modelscited by 1
  22. 2026Reversing the Flow: Generation-to-Understanding Synergy in Large Multimodal Modelscited by 1
  23. 2026Sample-Efficient Post-Training for LEGO Spatial-Physics Reasoningcited by 1
  24. 2026JigShape: Evaluating Visual-Geometric Reasoning in VLMs through Jigsaw Puzzlescited by 1
  25. 2026Large Language Models and their Awareness of Mechanics and Spatial Geometrycited by 1
  26. 2026Rule-Compliant Visual Spatial Planning for Multimodal Large Language Modelscited by 1
  27. 2025Ascending the Infinite Ladder: Benchmarking Spatial Deformation Reasoning in Vision-Language Modelscited by 0
  28. 2025Prompt-to-Parts: Generative AI for Physical Assembly and Scalable Instructionscited by 0
  29. 2026VisionFoundry: Teaching VLMs Visual Perception with Synthetic Imagescited by 0
  30. 2026Spatial Competence Benchmarkcited by 0
  31. 2026Flat-Pack Bench: Evaluating Spatio-Temporal Understanding in Large Vision-Language Models through Furniture Assemblycited by 0
  32. 2026VGenST-Bench: A Benchmark for Spatio-Temporal Reasoning via Active Video Synthesiscited by 0
  33. 2026Brick-Composer: Using MLLMs for Assembly with Diverse Brickscited by 0
  34. 2026Seeing Time: Benchmarking Chronological Reasoning and Shortcut Biases in Vision-Language Modelscited by 0
  35. 2026The Periodic Table of LLM Reasoning: A Structured Survey of Reasoning Paradigms, Methods, and Failure Modescited by 0
  36. 2026SpaR3D-MoE: Adaptive 3D Spatial Reasoning from Sparse Views Meets Geometry-Inductive Mixture-of-Expertscited by 0
  37. 2026PolyBridgeBench: Benchmarking Multimodal LLMs for Physics-Grounded Bridge Designcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Scaling and Beyond: Advancing Spatial Reasoning in MLLMs Requires New Recipes
  2. 2025Spatial Reasoning in Multimodal Large Language Models: A Survey of Tasks, Benchmarks and Methods
  3. 2025SIRI-Bench: Challenging VLMs' Spatial Intelligence through Complex Reasoning Tasks
  4. 2025MARBLE: A Hard Benchmark for Multimodal Spatial Reasoning and Planning
  5. 2025Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarks
  6. 2026EgoMind: Activating Spatial Cognition through Linguistic Reasoning in MLLMs
  7. 2026Do LLMs Build Spatial World Models? Evidence from Grid-World Maze Tasks
  8. 2025SpatialBench: Benchmarking Multimodal Large Language Models for Spatial Cognition