mlsanityData to 7 Oct 2026

Simulation to Rules: A Dual-VLM Framework for Formal Visual Planning

arXiv 2510.03182 ↗First posted 3 Oct 2025cs.RO cs.AI cs.CLcited by 4

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Vision Language Models (VLMs) show strong potential for visual planning but struggle with precise spatial and long-horizon reasoning, while Planning Domain Definition Language (PDDL) planners excel at formal long-horizon planning but cannot interpret visual inputs. Recent works combine these complementary advantages by translating visual problems into PDDL. However, while VLMs can generate PDDL problem files satisfactorily, accurately generating PDDL domain files, which encode planning rules, remains challenging and typically requires human expertise or environment interaction. We propose VLMFP, a Dual-VLM-guided framework that autonomously generates both PDDL problem and domain files for formal visual planning. VLMFP combines a SimVLM that simulates action consequences with a GenVLM that generates and iteratively refines PDDL files by aligning symbolic execution with simulated outcomes, enabling multiple levels of generalization across unseen instances, visual appearances, and game rules. We evaluate VLMFP on 6 grid-world domains and demonstrate its generalization capability. On average, SimVLM achieves 87.3% and 86.0% scenario understanding and action simulation for seen and unseen appearances, respectively. With the guidance of SimVLM, VLMFP attains 70.0%, 54.1% planning success on unseen instances in seen and unseen appearances, respectively. We further demonstrate that VLMFP scales to complex long-horizon 3D planning tasks, including multi-robot collaboration and assembly scenarios with partial observability and diverse visual variations. Project page: https://sites.google.com/view/vlmfp.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 47 references that are in the map. First posted on arXiv 3 Oct 2025.

Builds on

32 of 47 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models27 Mar
  2. 2025Reflective Planning: Vision-Language Models for Multi-Stage Long-Horizon Robotic Manipulation23 Feb
  3. 2025Planning with Vision-Language Models and a Use Case in Robot-Assisted Teaching29 Jan
  4. 2024VSP: Assessing the dual challenges of perception and reasoning in spatial planning tasks for VLMs2 Jul
  5. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs12 Feb
  6. 2023Grounding Classical Task Planners via Vision-Language Models17 Apr
  7. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar

Other references · 25

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  5. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  6. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  7. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 496
  8. 2023Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoningcited by 379
  9. 2023On the Planning Abilities of Large Language Models : A Critical Investigationcited by 308
  10. 2022Autoformalization with Large Language Modelscited by 252
  11. 2023Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planningcited by 204
  12. 2024PaliGemma 2: A Family of Versatile VLMs for Transfercited by 204
13 more
  1. 2025Imagine while Reasoning in Space: Multimodal Visualization-of-Thoughtcited by 204
  2. 2024LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworkscited by 178
  3. 2023Translating Natural Language to Planning Goals with Large-Language Modelscited by 132
  4. 2023Solving Math Word Problems by Combining Language Models With Symbolic Solverscited by 124
  5. 2023SatLM: Satisfiability-Aided Language Models Using Declarative Promptingcited by 100
  6. 2023Large Language Models for Supply Chain Optimizationcited by 61
  7. 2020PDDLGym: Gym Environments from PDDL Problemscited by 45
  8. 2024Formal-LLM: Integrating Formal Language and Natural Language for Controllable LLM-based Agentscited by 39
  9. 2024Planning Anything with Rigor: General-Purpose Zero-Shot Planning with LLM-based Formalized Programmingcited by 35
  10. 2023Vision-Language Interpreter for Robot Task Planningcited by 29
  11. 2023Mini-BEHAVIOR: A Procedurally Generated Benchmark for Long-horizon Decision-Making in Embodied AIcited by 23
  12. 2024Leveraging Environment Interaction for Automated PDDL Translation and Planning with Large Language Modelscited by 17
  13. 2025Collision- and Reachability-Aware Multi-Robot Control with Grounded LLM Plannerscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Learning Visual Spatial Planning from Symbolic State via Modality-Gap-Aware Self-Distillationcited by 1
  2. 2026PDDL-ART: Autonomous Symbolic Abstraction From Demonstration For Long-Horizon Robotic Manipulation Using Vision-Language Modelscited by 0
  3. 2026Resolving State-Representation Mismatch: State-Space Visual Reasoning for Open-Loop VLA Planningcited by 0
  4. 2026Bayesian Active Learning for Intent Disambiguation in Interactive Robot Planningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Vision Language Models Cannot Plan, but Can They Formalize?
  2. 2025ViPlan: A Benchmark for Visual Planning with Symbolic Predicates and Vision-Language Models
  3. 2024DKPROMPT: Domain Knowledge Prompting Vision-Language Models for Open-World Planning
  4. 2024Evaluating Vision-Language Models as Evaluators in Path Planning
  5. 2025Visual Planning: Let's Think Only with Images
  6. 2025Planning with Reasoning using Vision Language World Model
  7. 2025VLMPlanner: Integrating Visual Language Models with Motion Planning
  8. 2025Making VLMs More Robot-Friendly: Self-Critical Distillation of Low-Level Procedural Reasoning