mlsanityData to 13 Oct 2026

VSP: Assessing the dual challenges of perception and reasoning in spatial planning tasks for VLMs

arXiv 2407.01863 ↗First posted 2 Jul 2024cs.CLcited by 47

On the map: LLM reasoning → Spatial and embodied reasoning

also in 1 more
Abstract (arXiv)

Vision language models (VLMs) are an exciting emerging class of language models (LMs) that have merged classic LM capabilities with those of image processing systems. However, the ways that these capabilities combine are not always intuitive and warrant direct investigation. One understudied capability in VLMs is visual spatial planning -- the ability to comprehend the spatial arrangements of objects and devise action plans to achieve desired outcomes in visual scenes. In our study, we introduce VSP, a benchmark that 1) evaluates the spatial planning capability in these models in general, and 2) breaks down the visual planning task into finer-grained sub-tasks, including perception and reasoning, and measure the LMs capabilities in these sub-tasks. Our evaluation shows that both open-source and private VLMs fail to generate effective plans for even simple spatial planning tasks. Evaluations on the fine-grained analytical tasks further reveal fundamental deficiencies in the models' visual perception and bottlenecks in reasoning abilities, explaining their worse performance in the general spatial planning tasks. Our work illuminates future directions for improving VLMs' abilities in spatial planning. Our benchmark is publicly available at https://github.com/UCSB-NLP-Chang/Visual-Spatial-Planning.

Key facts. Cited by 47 papers in the map (data through 13 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 40 of its 68 references that are in the map. First posted on arXiv 2 Jul 2024.

Builds on

40 of 68 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Can Large Language Models be Good Path Planners? A Benchmark and Investigation on Spatial-temporal Reasoning5 Oct

Other references · 39

  1. 2023GPT-4 Technical Reportcited by 18,278
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,502
  4. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,321
  5. 2024Mixtral of Expertscited by 1,811
  6. 2016OpenAI Gymcited by 1,694
  7. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,637
  8. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,230
  9. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 933
  10. 2023Reasoning with Language Model is Planning with World Modelcited by 851
  11. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 848
  12. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 723
27 more
  1. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 706
  2. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 680
  3. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 634
  4. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 577
  5. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 563
  6. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Datasetcited by 521
  7. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 488
  8. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agentscited by 386
  9. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 377
  10. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  11. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 285
  12. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGIcited by 193
  13. 2024LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworkscited by 179
  14. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
  15. 2024On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Taskscited by 140
  16. 2023SmartPlay: A Benchmark for LLMs as Intelligent Agentscited by 124
  17. 2023Dolphins: Multimodal Language Model for Drivingcited by 124
  18. 2024MileBench: Benchmarking MLLMs in Long Contextcited by 86
  19. 2023TPTU: Large Language Model-based AI Agents for Task Planning and Tool Usagecited by 55
  20. 2024Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Modelscited by 53
  21. 2023CoPAL: Corrective Planning of Robot Actions with Large Language Modelscited by 40
  22. 2023MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteriacited by 27
  23. 2023Gemini vs GPT-4V: A Preliminary Comparison and Combination of Vision-Language Models Through Qualitative Casescited by 12
  24. 2023LGMCTS: Language-Guided Monte-Carlo Tree Search for Executable Semantic Object Rearrangementcited by 11
  25. 2024Visually Dehallucinative Instruction Generation: Know What You Don't Knowcited by 10
  26. 2023From Cooking Recipes to Robot Task Trees -- Improving Planning Correctness and Task Efficiency by Leveraging LLMs with a Knowledge Networkcited by 9
  27. 2019Blocksworld Revisited: Learning and Reasoning to Generate Event-Sequences from Image Pairscited by 1

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 2; 2025: 19; 2026: 26.

47 papers.

Sort:

By area:

  1. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 197
  2. 2025Machine Mental Imagery: Empower Multimodal Reasoning with Latent Visual Tokenscited by 122
  3. 2025Visual Planning: Let's Think Only with Imagescited by 85
  4. 2025ThinkMorph: Emergent Properties in Multimodal Interleaved Chain-of-Thought Reasoningcited by 56
  5. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 42
  6. 2025Are Video Models Ready as Zero-Shot Reasoners? An Empirical Study with the MME-CoF Benchmarkcited by 41
  7. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 39
  8. 2025SpatialBench: Benchmarking Multimodal Large Language Models for Spatial Cognitioncited by 35
  9. 2025iVISPAR -- An Interactive Visual-Spatial Reasoning Benchmark for VLMscited by 25
  10. 2024Multimodal Self-Instruct: Synthetic Abstract Image and Visual Reasoning Instruction Using Language Modelcited by 21
  11. 2025Agent-RewardBench: Towards a Unified Benchmark for Reward Modeling across Perception, Planning, and Safety in Real-World Multimodal Agentscited by 20
  12. 2025Are Large Vision Language Models Good Game Players?cited by 19
35 more
  1. 2025DiffThinker: Towards Generative Multimodal Reasoning with Diffusion Modelscited by 18
  2. 2026Visual-OPSD: Cross-Modal On-Policy Self-Distillation for Efficient Unified Multimodal Reasoningcited by 15
  3. 2025Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?cited by 13
  4. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  5. 2026LatentUM: Unleashing the Potential of Interleaved Cross-Modal Reasoning via a Latent-Space Unified Modelcited by 7
  6. 2026MentisOculi: Revealing the Limits of Reasoning with Mental Imagerycited by 6
  7. 2026What's Holding Back Latent Visual Reasoning?cited by 6
  8. 2024Bridging the Visual Gap: Fine-Tuning Multimodal Models with Knowledge-Adapted Captionscited by 5
  9. 2025ViPlan: A Benchmark for Visual Planning with Symbolic Predicates and Vision-Language Modelscited by 5
  10. 2025LRR-Bench: Left, Right or Rotate? Vision-Language models Still Struggle With Spatial Understanding Taskscited by 5
  11. 2026AdaReasoner: Dynamic Tool Orchestration for Iterative Visual Reasoningcited by 5
  12. 2025Simulation to Rules: A Dual-VLM Framework for Formal Visual Planningcited by 4
  13. 2026PerceptionComp: A Video Benchmark for Complex Perception-Centric Reasoningcited by 3
  14. 2025RCI: A Score for Evaluating Global and Local Reasoning in Multimodal Benchmarkscited by 2
  15. 2025Downscaling Intelligence: Exploring Perception and Reasoning Bottlenecks in Small Multimodal Modelscited by 2
  16. 2026Learning Visual Spatial Planning from Symbolic State via Modality-Gap-Aware Self-Distillationcited by 2
  17. 2026Scaffolding Minds: Optimizing Latent Visual Target Representations for Multimodal Reasoningcited by 2
  18. 2025CoSPlan: Corrective Sequential Planning via Scene Graph Incremental Updatescited by 1
  19. 2026On the Out-of-Distribution Generalization of Reasoning in Multimodal LLMs for Simple Visual Planning Taskscited by 1
  20. 2026The Thinking Pixel: Recursive Sparse Reasoning in Multimodal Diffusion Latentscited by 1
  21. 2026DeepLatent: Think with Images via Parallel Latent Visual Reasoningcited by 1
  22. 2026Spectral-Progressive Thought Flow for Lightweight Multimodal Reasoningcited by 1
  23. 2026IMBench: A Benchmark for Intuitive Robotic Manipulationcited by 1
  24. 2026When Memory Lies: An Empirical Study of Spatial Memory Staleness in VLM Agentscited by 1
  25. 2026ItinBench: Benchmarking Planning Across Multiple Cognitive Dimensions with Large Language Modelscited by 0
  26. 2026STORM: Internalized Modeling for Spatial-Temporal Reasoning in Video-Language Modelscited by 0
  27. 2026InterSketch: An Interleaved Reasoning Model with Self-correcting Visual Sketch and Stepwise Rewardcited by 0
  28. 2026UniCanvas: A Diffusion-base Unified Model for Text-in-Image Joint Generationcited by 0
  29. 2026SCOPE: Evolving Symbolic World for Planning in Open-Ended Environmentscited by 0
  30. 2026Visually Grounded Self-Reflection for Vision-Language Models via Reinforcement Learningcited by 0
  31. 2026MindTopo: Can Foundation Models Reason in Topological Space?cited by 0
  32. 2026Resolving State-Representation Mismatch: State-Space Visual Reasoning for Open-Loop VLA Planningcited by 0
  33. 2026WM-VLM: Probing Internal World Models for Interleaved Visual-Textual Reasoningcited by 0
  34. 2026SpatialCORE: Confidence-Aware Grounded Spatial Reasoning in Large Vision--Language Modelscited by 0
  35. 2026Flow-of-Thought: A Framework for Visual Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Is Visual Prompting All You Need? Studying VLM Spatial Reasoning under Progressive Visual Scaffolds
  2. 2024Evaluating Vision-Language Models as Evaluators in Path Planning
  3. 2025Imagine in Space: Exploring the Frontier of Spatial Intelligence and Reasoning Efficiency in Vision Language Models
  4. 2026PlanBench-V: A Spatial Planning Map Benchmark for Vision-Language Models
  5. 2025Mind the Gap: Benchmarking Spatial Reasoning in Vision-Language Models
  6. 2026KilometerVision: A New Frontier for Large-Scale Spatial Intelligence in VLMs
  7. 2025Reasoning Path and Latent State Analysis for Multi-view Visual Spatial Reasoning: A Cognitive Science Perspective
  8. 2024Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models