mlsanityData to 14 Oct 2026

Visual Planning: Let's Think Only with Images

arXiv 2505.11409 ↗First posted 16 May 2025cs.LG cs.AI cs.CLcited by 85

On the map: LLM reasoning → Spatial and embodied reasoning

Abstract (arXiv)

Recent advancements in Large Language Models (LLMs) and their multimodal extensions (MLLMs) have substantially enhanced machine reasoning across diverse tasks. However, these models predominantly rely on pure text as the medium for both expressing and structuring reasoning, even when visual information is present. In this work, we argue that language may not always be the most natural or effective modality for reasoning, particularly in tasks involving spatial and geometrical information. Motivated by this, we propose a new paradigm, Visual Planning, which enables planning through purely visual representations for these "vision-first" tasks, as a supplementary channel to language-based reasoning. In this paradigm, planning is executed via sequences of images that encode step-by-step inference in the visual domain, akin to how humans sketch or visualize future actions. We introduce a novel reinforcement learning framework, Visual Planning via Reinforcement Learning (VPRL), empowered by GRPO for post-training large vision models, leading to substantial improvements in planning in a selection of representative visual navigation tasks, FrozenLake, Maze, and MiniBehavior. Our visual planning paradigm outperforms all other planning variants that conduct reasoning in the text-only space. Our results establish Visual Planning as a viable and promising supplement to language-based reasoning, opening new avenues for tasks that benefit from intuitive, image-based inference.

Key facts. Cited by 85 papers in the map (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 51 of its 71 references that are in the map. First posted on arXiv 16 May 2025.

Builds on

51 of 71 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024VSP: Assessing the dual challenges of perception and reasoning in spatial planning tasks for VLMs2 Jul
  2. 2024TopViewRS: Vision-Language Models as Top-View Spatial Reasoners4 Jun
  3. 2022Visual Spatial Reasoning30 Apr

Other references · 48

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  6. 2025Qwen2.5-VL Technical Reportcited by 5,587
  7. 2024GPT-4o System Cardcited by 4,588
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  9. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  10. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  11. 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
36 more
  1. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  2. 2016OpenAI Gymcited by 1,736
  3. 2018World Modelscited by 1,513
  4. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  5. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  6. 2023PaLM 2 Technical Reportcited by 1,162
  7. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  8. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  9. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  10. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 752
  11. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 672
  12. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Modelcited by 526
  13. 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 513
  14. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 452
  15. 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimizationcited by 301
  16. 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Modelscited by 268
  17. 2023Sequential Modeling Enables Scalable Learning for Large Vision Modelscited by 212
  18. 2025Imagine while Reasoning in Space: Multimodal Visualization-of-Thoughtcited by 206
  19. 2025T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoTcited by 166
  20. 2025R1-Zero's "Aha Moment" in Visual Reasoning on a 2B Non-SFT Modelcited by 165
  21. 2025Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmarkcited by 134
  22. 2025Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Stepcited by 131
  23. 2024M3CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thoughtcited by 121
  24. 2025Perception-R1: Pioneering Perception Policy with Reinforcement Learningcited by 120
  25. 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 117
  26. 2024ANOLE: An Open, Autoregressive, Native Large Multimodal Models for Interleaved Image-Text Generationcited by 114
  27. 2025SimpleAR: Pushing the Frontier of Autoregressive Visual Generation through Pretraining, SFT, and RLcited by 86
  28. 2024Image-of-Thought Prompting for Visual Reasoning Refinement in Multimodal Large Language Modelscited by 65
  29. 2025ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Modelscited by 58
  30. 2024Scaffolding Coordinates to Promote Vision-Language Coordination in Large Multi-Modal Modelscited by 57
  31. 2024Don't Look Twice: Faster Video Transformers with Run-Length Tokenizationcited by 43
  32. 2023A Configurable Library for Generating and Manipulating Maze Datasetscited by 31
  33. 2024CoMT: A Novel Benchmark for Chain of Multi-modal Thought on Large Vision-Language Modelscited by 30
  34. 2020Recurrent neural circuits for contour detectioncited by 23
  35. 2023Mini-BEHAVIOR: A Procedurally Generated Benchmark for Long-horizon Decision-Making in Embodied AIcited by 23
  36. 2024Semantic Map-based Generation of Navigation Instructionscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 38; 2026: 47.

85 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  2. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 199
  3. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  4. 2025Latent Visual Reasoningcited by 104
  5. 2025Zebra-CoT: A Dataset for Interleaved Vision Language Reasoningcited by 84
  6. 2025Monet: Reasoning in Latent Visual Space Beyond Images and Languagecited by 82
  7. 2025Look-Back: Implicit Visual Re-focusing in MLLM Reasoningcited by 55
  8. 2025Ground-R1: Incentivizing Grounded Visual Reasoning via Reinforcement Learningcited by 49
  9. 2025Latent Sketchpad: Sketching Visual Thoughts to Elicit Multimodal Reasoning in MLLMscited by 43
  10. 2025MM-BrowseComp: A Comprehensive Benchmark for Multimodal Browsing Agentscited by 42
  11. 2025VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Modelscited by 37
  12. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 35
38 more
  1. 2025JarvisEvo: Towards a Self-Evolving Photo Editing Agent with Synergistic Editor-Evaluator Optimizationcited by 31
  2. 2025From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Modelscited by 26
  3. 2025Sketch-in-Latents: Eliciting Unified Reasoning in MLLMscited by 25
  4. 2025A Survey on Agentic Multimodal Large Language Modelscited by 23
  5. 2025Reasoning in the Dark: Interleaved Vision-Text Reasoning in Latent Spacecited by 23
  6. 2025Reasoning via Video: The First Evaluation of Video Models' Reasoning Abilities through Maze-Solving Taskscited by 22
  7. 2025DiffThinker: Towards Generative Multimodal Reasoning with Diffusion Modelscited by 19
  8. 2026Visual Generation Unlocks Human-Like Reasoning through Multimodal World Modelscited by 19
  9. 2025On the Faithfulness of Visual Thinking: Measurement and Enhancementcited by 17
  10. 2026LensWalk: Agentic Video Understanding by Planning How You See in Videoscited by 17
  11. 2026Demystifying Video Reasoningcited by 14
  12. 2025ROVER: Benchmarking Reciprocal Cross-Modal Reasoning for Omnimodal Generationcited by 13
  13. 2025ARM-Thinker: Reinforcing Multimodal Generative Reward Models with Agentic Tool Use and Visual Reasoningcited by 13
  14. 2025DraCo: Draft as CoT for Text-to-Image Preview and Rare Concept Generationcited by 13
  15. 2025VR-Thinker: Boosting Video Reward Models through Thinking-with-Image Reasoningcited by 12
  16. 2026GUI-Eyes: Tool-Augmented Perception for Visual Grounding in GUI Agentscited by 10
  17. 2026Thinking in Frames: How Visual Context and Test-Time Scaling Empower Video Reasoningcited by 10
  18. 2025Reinforcement Learning for Large Model: A Surveycited by 9
  19. 2025MM-R1: Unleashing the Power of Unified Multimodal Large Language Models for Personalized Image Generationcited by 9
  20. 202511Plus-Bench: Demystifying Multimodal LLM Spatial Reasoning with Cognitive-Inspired Analysiscited by 9
  21. 2025Octopus: Agentic Multimodal Reasoning with Six-Capability Orchestrationcited by 9
  22. 2026Rethinking Token-Level Policy Optimization for Multimodal Chain-of-Thoughtcited by 9
  23. 2026Seeing Is Believing? A Benchmark for Multimodal Large Language Models on Visual Illusions and Anomaliescited by 8
  24. 2026SketchVL: Policy Optimization via Fine-Grained Credit Assignment for Chart Understanding and Morecited by 7
  25. 2025Beyond Textual CoT: Interleaved Text-Image Chains with Deep Confidence Reasoning for Image Editingcited by 6
  26. 2026VBVR-Pro: A Scalable and Verifiable Suite for Native Visual Reasoningcited by 6
  27. 2025Agentic Systems in Radiology: Design, Applications, Evaluation, and Challengescited by 5
  28. 2025Boosting Reasoning in Large Multimodal Models via Activation Replaycited by 5
  29. 2026Improving Visual Reasoning with Iterative Evidence Refinementcited by 5
  30. 2025Thinking with Images via Self-Calling Agentcited by 4
  31. 2026VisualThink-VLA: Visual Intermediate Reasoning for Effective and Low-Latency Vision-Language-Action Policiescited by 4
  32. 2025Bridging Formal Language with Chain-of-Thought Reasoning to Geometry Problem Solvingcited by 3
  33. 2025edgeVLM: Cloud-edge Collaborative Real-time VLM based on Context Transfercited by 3
  34. 2026DuoGen: Towards General Purpose Interleaved Multimodal Generationcited by 3
  35. 2026Visual Enhanced Depth Scaling for Multimodal Latent Reasoningcited by 3
  36. 2026Self-Consistent Latent Reasoning: Long Latent Sequence Reasoning for Vision-Language Modelcited by 3
  37. 2026Grounded 3D-Aware Spatial Vision-Language Modelingcited by 3
  38. 2026OpenCoF: Learning to Reason Through Video Generationcited by 3

50 most cited of 85 All 85 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Visually-Guided Policy Optimization for Multimodal Reasoning
  2. 2025Learning Only with Images: Visual Reinforcement Learning with Reasoning, Rendering, and Visual Feedback
  3. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planning
  4. 2025More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Models
  5. 2025World-aware Planning Narratives Enhance Large Vision-Language Model Planner
  6. 2025UNeMo: Collaborative Visual-Language Reasoning and Navigation via a Multimodal World Model
  7. 2025Assessing the Value of Visual Input: A Benchmark of Multimodal Large Language Models for Robotic Path Planning
  8. 2025Planning with Reasoning using Vision Language World Model