mlsanityData to 14 Oct 2026

Planning with Reasoning using Vision Language World Model

arXiv 2509.02722 ↗First posted 2 Sep 2025cs.AIcited by 36

On the map: World models → Embodied and robot world models

also in 1 more
Abstract (arXiv)

Effective planning requires strong world models, but high-level world models that can understand and reason about actions with semantic and temporal abstraction remain largely underdeveloped. We introduce the Vision Language World Model (VLWM), a foundation model trained for language-based world modeling on natural videos. Given visual observations, the VLWM first infers the overall goal achievements then predicts a trajectory composed of interleaved actions and world state changes. Those targets are extracted by iterative LLM Self-Refine conditioned on compressed future observations represented by Tree of Captions. The VLWM learns both an action policy and a dynamics model, which respectively facilitates reactive system-1 plan decoding and reflective system-2 planning via cost minimization. The cost evaluates the semantic distance between the hypothetical future states given by VLWM roll-outs and the expected goal state, and is measured by a critic model that we trained in a self-supervised manner. The VLWM achieves state-of-the-art Visual Planning for Assistance (VPA) performance on both benchmark evaluations and our proposed PlannerArena human evaluations, where system-2 improves the Elo score by +27% upon system-1. The VLWM models also outperforms strong VLM baselines on RoboVQA and WorldPrediction benchmark.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). Placed in Embodied and robot world models, one of 11 directions in World models. Builds on 74 of its 84 references that are in the map. First posted on arXiv 2 Sep 2025.

Builds on

74 of 84 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning11 Jun
  2. 2025WorldPrediction: A Benchmark for High-level World Modeling and Long-horizon Procedural Planning4 Jun
  3. 20243D-VLA: A 3D Vision-Language-Action Generative World Model14 Mar

Other references · 71

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  7. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  8. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  9. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  10. 2018World Modelscited by 1,513
  11. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
  12. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,261
38 more
  1. 2023Mastering Diverse Domains through World Modelscited by 1,253
  2. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  3. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  4. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  5. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  6. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 881
  7. 2023Large Language Models as Optimizerscited by 864
  8. 2025Cosmos World Foundation Model Platform for Physical AIcited by 864
  9. 2023Reasoning with Language Model is Planning with World Modelcited by 861
  10. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 795
  11. 2024Genie: Generative Interactive Environmentscited by 747
  12. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 637
  13. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 619
  14. 2020Rescaling Egocentric Visioncited by 540
  15. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 531
  16. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 490
  17. 2023Learning Interactive Real-World Simulatorscited by 458
  18. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  19. 2024DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planningcited by 386
  20. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 347
  21. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 345
  22. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 338
  23. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 326
  24. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Drivingcited by 319
  25. 2019COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysiscited by 318
  26. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 273
  27. 2024Navigation World Modelscited by 272
  28. 2019Cross-task weakly supervised learning from instructional videoscited by 199
  29. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 199
  30. 2025ThinkAct: Vision-Language-Action Reasoning via Reinforced Visual Latent Planningcited by 156
  31. 2024Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agentscited by 154
  32. 2025RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concretecited by 135
  33. 2024Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigationcited by 126
  34. 2023RoboVQA: Multimodal Long-Horizon Reasoning for Roboticscited by 120
  35. 2019Imitation Learning from Imperfect Demonstrationcited by 114
  36. 2023Motion Planning for Autonomous Driving: The State of the Art and Future Perspectivescited by 114
  37. 2024WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environmentcited by 107
  38. 2024RoboMamba: Efficient Vision-Language-Action Model for Robotic Reasoning and Manipulationcited by 103

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 11; 2026: 25.

36 papers.

Sort:

By area:

  1. 2025World Simulation with Video Foundation Models for Physical AIcited by 200
  2. 2025VL-JEPA: Joint Embedding Predictive Architecture for Vision-languagecited by 68
  3. 2025Scaling Agent Learning via Experience Synthesiscited by 44
  4. 2026Beyond Language Modeling: An Exploration of Multimodal Pretrainingcited by 37
  5. 2026Visual Generation Unlocks Human-Like Reasoning through Multimodal World Modelscited by 19
  6. 2025MobileWorldBench: Towards Semantic World Modeling For Mobile Agentscited by 18
  7. 2025A Step Toward World Models: A Survey on Robotic Manipulationcited by 15
  8. 2026Code as Worlds: Agentic Discovery of Executable World Representations for Physical Reasoningcited by 14
  9. 2026Action100M: A Large-scale Video Action Datasetcited by 12
  10. 2026Egocentric World Model for Photorealistic Hand-Object Interaction Synthesiscited by 12
  11. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  12. 2026GeoWorld: Geometric World Modelscited by 6
24 more
  1. 2025Building Egocentric Procedural AI Assistant: Methods, Benchmarks, and Challengescited by 5
  2. 2025Open-Ended Goal Inference through Actions and Language for Human-Robot Collaborationcited by 5
  3. 2026Self-Improving World Modelling with Latent Actionscited by 5
  4. 2026Anticipatory Planning for Multimodal AI Agentscited by 4
  5. 2026Learning Visual Feature-Based World Models via Residual Latent Actioncited by 4
  6. 2025TV2TV: A Unified Framework for Interleaved Language and Video Generationcited by 3
  7. 2026World Models: A Comprehensive Survey of Architectures, Methodologies, Reasoning Paradigms, and Applicationscited by 3
  8. 2026OSEF: One-Step Evidence Fusion for Cross-Video Scene Procedure Planningcited by 3
  9. 2026Beyond Simply Environment Scaling: Designing Effective Environment Distributions for Multimodal Agent Learningcited by 3
  10. 2025X-WIN: Building Chest Radiograph World Model via Predictive Sensingcited by 2
  11. 2026Reward Prediction with Factorized World Statescited by 2
  12. 2025Concepts in Motion: Temporal Concept Bottleneck Model for Interpretable Video Classificationcited by 1
  13. 2025Audio-Visual World Models: Learning Physically Grounded Multisensory Dynamicscited by 1
  14. 2026Self-supervised Hierarchical Visual Reasoning with World Modelcited by 1
  15. 2026How You Move Tells What You'll Do: Trajectory-Conditioned Egocentric Predictioncited by 1
  16. 2026MATEO: A Multimodal Benchmark for Temporal Reasoning and Planning in LVLMscited by 0
  17. 2026SiMing-Bench: Evaluating Procedural Correctness from Continuous Interactions in Clinical Skill Videoscited by 0
  18. 2026ICAT: Incident-Case-Grounded Adaptive Testing for Physical-Risk Prediction in Embodied World Modelscited by 0
  19. 2026RECIPE: Procedural Planning via Grounding in Instructional Videocited by 0
  20. 2026RoboTALES: Learning Reasoning-Guided Robot Policies via Task-Aligned Simulated Futurescited by 0
  21. 2026Grounding Spatial Relations in a Compact World Model: Instruction Leakage and a Goal-Free Dynamics Fixcited by 0
  22. 2026WMLLM: Self-Evolving Optimization Agents via Predict-Then-Act World Modelingcited by 0
  23. 2026DroneWAM: Efficient World Action Model for Drone Visual Navigationcited by 0
  24. 2026VIDEAS: Distilling Explicit Action Semantics from Demonstration Videos for World Models via Prior-Guided Simulationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025World-aware Planning Narratives Enhance Large Vision-Language Model Planner
  2. 2026World Action Planner: Generalizable Decision-Making with Action-Conditioned World Models
  3. 2026World-Value-Action Model: Implicit Planning for Vision-Language-Action Systems
  4. 2025Visual Planning: Let's Think Only with Images
  5. 2025Semantic World Models
  6. 2024Evaluating Vision-Language Models as Evaluators in Path Planning
  7. 2026Explicit Language Memory for Long-Horizon Planning in Vision-Language-Action Models
  8. 2026World2VLM: Distilling World Model Imagination into VLMs for Dynamic Spatial Reasoning