mlsanityData to 14 Oct 2026

Write Once, Plan Many: Building Reusable Visual Planning Systems from One Scene

arXiv 2610.19234 ↗First posted 13 Oct 2026cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Vision-language models (VLMs) can reason about visual planning tasks, but querying them for every instance incurs repeated inference cost and latency. Generating reusable planning programs offers an alternative, but constructing reliable planning systems for new problem domains without extensive supervision remains challenging. We present a training-free framework that constructs reusable perception and planning programs from a domain specification and a single practice scene without a paired solution. The central challenge is obtaining useful feedback from the limited input. Our framework addresses it by leveraging VLM's ability to provide a renderer and a dynamic model based on its scene understanding. Rendering detected states enables comparison with the practice scene, while checking simulated action outcomes against the domain specification helps expose errors in rule implementation. Together, these checks guide refinement of the grounding program and planner without requiring solution labels. After refinement and component selection, the programs are frozen and solve new instances without further VLM calls. The same framework constructs systems across domains without manual pipeline redesign. Across fifteen domain variants spanning 2D and 3D scenes with discrete and continuous state, our systems achieve 99.3% average planning success, compared with 80.9% for the strongest domain-level program-generation baseline and 90.7% for the strongest instance-level baseline. Cost analyses show a favorable success-cost trade-off, and ablations confirm the contributions of the executable domain model, execution feedback, and component selection.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 28 references that are in the map. First posted on arXiv 13 Oct 2026.

Builds on

27 of 28 references.

Closest ancestors · 8

Same direction, newest first

  1. 2026Sandboxed Coding Agents are Competitive Omni-modal Task Solvers30 May
  2. 2025Simulation to Rules: A Dual-VLM Framework for Formal Visual Planning3 Oct
  3. 2025Vision Language Models Cannot Plan, but Can They Formalize?25 Sep
  4. 2025Reflective Planning: Vision-Language Models for Multi-Stage Long-Horizon Robotic Manipulation23 Feb
  5. 2025Planning with Vision-Language Models and a Use Case in Robot-Assisted Teaching29 Jan
  6. 2024RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics15 Jun
  7. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs12 Feb
  8. 2023GPT-4V(ision) for Robotics: Multimodal Task Planning from Human Demonstration20 Nov

Other references · 19

  1. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  2. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  3. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 1,002
  4. 2023Teaching Large Language Models to Self-Debugcited by 988
  5. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  6. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 358
  7. 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Modelscited by 268
  8. 2023LEVER: Learning to Verify Language-to-Code Generation with Executioncited by 239
  9. 2024WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environmentcited by 107
  10. 2025VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Usecited by 90
  11. 2022Parsel: Algorithmic Reasoning with Language Models by Composing Decompositionscited by 64
  12. 2024CodeTree: Agent-guided Tree Search for Code Generation with Large Language Modelscited by 34
7 more
  1. 2023Vision-Language Interpreter for Robot Task Planningcited by 30
  2. 2025CodeV: Code with Images for Faithful Visual Reasoning via Tool-Aware Policy Optimizationcited by 26
  3. 2024Wonderful Team: Zero-Shot Physical Task Planning with Visual LLMscited by 11
  4. 2026Tycho: Active Abstraction with Programmatic World Models for ARC-AGI-3cited by 7
  5. 2026UniPlan: Vision-Language Task Planning for Mobile Manipulation with Unified PDDL Formulationcited by 4
  6. 2026Don't Show Pixels, Show Cues: Unlocking Visual Tool Reasoning in Language Models via Perception Programscited by 4
  7. 2025Draft and Refine with Visual Expertscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024VSP: Assessing the dual challenges of perception and reasoning in spatial planning tasks for VLMs
  2. 2025World-aware Planning Narratives Enhance Large Vision-Language Model Planner
  3. 2026Plan-Bench: Can Existing VLMs Go Well Beyond Perception to Reason and Plan in Embodied Worlds?
  4. 2026Resolving State-Representation Mismatch: State-Space Visual Reasoning for Open-Loop VLA Planning
  5. 2025Planning with Reasoning using Vision Language World Model
  6. 2025Making VLMs More Robot-Friendly: Self-Critical Distillation of Low-Level Procedural Reasoning
  7. 2024Evaluating Vision-Language Models as Evaluators in Path Planning
  8. 2026Dive into the Scene: Breaking the Perceptual Bottleneck in Vision-Language Decision Making via Focus Plan Generation