mlsanityData to 7 Oct 2026

WMPO: World Model-based Policy Optimization for Vision-Language-Action Models

arXiv 2511.09515 ↗First posted 12 Nov 2025cs.RO cs.AIcited by 68

On the map: World models → Embodied and robot world models

also in 1 more
Abstract (arXiv)

Vision-Language-Action (VLA) models have shown strong potential for general-purpose robotic manipulation, but their reliance on expert demonstrations limits their ability to learn from failures and perform self-corrections. Reinforcement learning (RL) addresses these through self-improving interactions with the physical environment, but suffers from high sample complexity on real robots. We introduce World-Model-based Policy Optimization (WMPO), a principled framework for on-policy VLA RL without interacting with the real environment. In contrast to widely used latent world models, WMPO focuses on pixel-based predictions that align the "imagined" trajectories with the VLA features pretrained with web-scale images. Crucially, WMPO enables the policy to perform on-policy GRPO that provides stronger performance than the often-used off-policy methods. Extensive experiments in both simulation and real-robot settings demonstrate that WMPO (i) substantially improves sample efficiency, (ii) achieves stronger overall performance, (iii) exhibits emergent behaviors such as self-correction, and (iv) demonstrates robust generalization and lifelong learning capabilities.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). Placed in Embodied and robot world models, one of 11 directions in World models. Builds on 30 of its 45 references that are in the map. First posted on arXiv 12 Nov 2025.

Builds on

30 of 45 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025World4RL: Diffusion World Models for Policy Refinement with Reinforcement Learning for Robotic Manipulation23 Sep
  2. 2024IRASim: A Fine-Grained World Model for Robot Manipulation20 Jun

Other references · 28

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  4. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  5. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  7. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  8. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  9. 2023Mastering Diverse Domains through World Modelscited by 1,193
  10. 2020Mastering Atari with Discrete World Modelscited by 990
  11. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  12. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Successcited by 844
16 more
  1. 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
  2. 2024Open-Sora: Democratizing Efficient Video Production for Allcited by 637
  3. 2025Flow-GRPO: Training Flow Matching Models via Online RLcited by 605
  4. 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 308
  5. 2019Understanding and Improving Layer Normalizationcited by 213
  6. 2025VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learningcited by 166
  7. 2024Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learningcited by 153
  8. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 149
  9. 2024Data Scaling Laws in Imitation Learning for Robotic Manipulationcited by 144
  10. 2025SimpleVLA-RL: Scaling VLA Training via Reinforcement Learningcited by 141
  11. 2025ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policycited by 139
  12. 2024GRAPE: Generalizing Robot Policy via Preference Alignmentcited by 106
  13. 2025Reinforcement Learning with Action Chunkingcited by 97
  14. 2025Horizon Reduction Makes RL Scalablecited by 55
  15. 2024Continuous Control with Coarse-to-fine Reinforcement Learningcited by 15
  16. 2025Human-assisted Robotic Policy Refinement via Action Preference Optimizationcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 65.

68 papers.

Sort:

By area:

  1. 2026DreamDojo: A Generalist Robot World Model from Large-Scale Human Videoscited by 117
  2. 2026InternVLA-A1: Unifying Understanding, Generation and Action for Robotic Manipulationcited by 48
  3. 2026WoVR: World Models as Reliable Simulators for Post-Training VLA Policies with RLcited by 46
  4. 2026World Action Models: The Next Frontier in Embodied AIcited by 46
  5. 2026World Model for Robot Learning: A Comprehensive Surveycited by 42
  6. 2025GR-RL: Going Dexterous and Precise for Long-Horizon Robotic Manipulationcited by 41
  7. 2026RISE: Self-Improving Robot Policy with Compositional World Modelcited by 39
  8. 2026VLAW: Iterative Co-Improvement of Vision-Language-Action Policy and World Modelcited by 39
  9. 2026World-VLA-Loop: Closed-Loop Learning of Video World Model and VLA Policycited by 36
  10. 2026World-Gymnast: Training Robots with Reinforcement Learning in a World Modelcited by 25
  11. 2026AtomVLA: Scalable Post-Training for Robotic Manipulation via Predictive Latent World Modelscited by 21
  12. 2026Reconstruction or Semantics? What Makes a Latent Space Useful for Robotic World Modelscited by 21
38 more
  1. 2026MultiWorld: Scalable Multi-Agent Multi-View Video World Modelscited by 19
  2. 2026Self-Improvements in Modern Agentic Systems: A Surveycited by 17
  3. 2026PlayWorld: Learning Robot World Models from Autonomous Playcited by 16
  4. 2026World-Value-Action Model: Implicit Planning for Vision-Language-Action Systemscited by 13
  5. 2026Towards Long-Lived Robots: Continual Learning VLA Models via Reinforcement Fine-Tuningcited by 11
  6. 2026STARRY: Spatial-Temporal Action-Centric World Modeling for Robotic Manipulationcited by 10
  7. 2025MIND-V: Hierarchical World Model for Long-Horizon Robotic Manipulation with RL-based Physical Alignmentcited by 9
  8. 2026RoboStereo: Dual-Tower 4D Embodied World Models for Unified Policy Optimizationcited by 9
  9. 2026How Should World Models Be Evaluated for Embodied Decision-Making? A Decision-Making-Centric Positioncited by 9
  10. 2026ProgressVLA: Progress-Guided Diffusion Policy for Vision-Language Robotic Manipulationcited by 8
  11. 2026WorldArena 2.0: Extending Embodied World Model Benchmarking on Modality, Functionality and Platformcited by 8
  12. 2026Point Tracking Improves World Action Modelscited by 7
  13. 2026Enhancing Policy Learning with World-Action Modelcited by 6
  14. 2026Motus2: A Self-Evolving General World Model for Dexterous Manipulationcited by 6
  15. 2026IG-RFT: An Interaction-Guided RL Framework for VLA Models in Long-Horizon Robotic Manipulationcited by 5
  16. 2026DreamPlan: Efficient Reinforcement Fine-Tuning of Vision-Language Planners via Video World Modelscited by 5
  17. 2025An Anatomy of Vision-Language-Action Models: From Modules to Milestones and Challengescited by 4
  18. 2026Towards Practical World Model-based Reinforcement Learning for Vision-Language-Action Modelscited by 4
  19. 2026Escaping the Diversity Trap in Robotic Manipulation via Anchor-Centric Adaptationcited by 4
  20. 2026Pre-VLA: Preemptive Runtime Verification for Reliable Vision-Language-Action and World-Model Rolloutscited by 4
  21. 2026Data Pyramid for Embodied Manipulation: A Surveycited by 4
  22. 2026Sword: Style-Robust World Models as Simulators via Dynamic Latent Bootstrapping for VLA Policy Post-Trainingcited by 3
  23. 2026Learning Action Manifold with Multi-view Latent Priors for Robotic Manipulationcited by 3
  24. 2026Learning Transferable Dynamics Priors from Action to World Modelingcited by 3
  25. 2026RedFlow: Redirect Failure into Action-Level Corrections for Flow-matching VLA Policycited by 3
  26. 2026Towards Generalist Game Players: An Investigation of Foundation Models in the Game Multiversecited by 2
  27. 2026PearlVLA: Progressive Embodied Action-Plan Refinement in Latent Spacecited by 2
  28. 2026SafeDojo: Safe Reinforcement Learning for VLA via Interactive World Modelcited by 2
  29. 2026WorldSample: Closed-loop Real-robot RL with World Modellingcited by 2
  30. 2026ExToken: Structured Exploration for Efficient Vision-Language-Action Reinforcement Fine-tuningcited by 2
  31. 2026GeniWorld: A Generalizable Interactive World Model for Robotic Manipulation via Visual Actionscited by 2
  32. 2026Q-Learning With World Modelscited by 2
  33. 2026OnlineWM: Causality-Aware Active Online Learning for Effective World Modelingcited by 2
  34. 2026World Models as an Intermediary between Agents and the Real Worldcited by 1
  35. 2026PVI: Plug-in Visual Injection for Vision-Language-Action Modelscited by 1
  36. 2026Blockchain Infrastructure for Intelligent Cyber--Physical--Social Systems:Post-Quantum Security, Interoperability, and Trustworthy Data Economies in the Era of Embodied AIcited by 1
  37. 2026TACO: TActile World Model as a Self-COrrector for Scalable Robot Policy Post-Trainingcited by 1
  38. 2026CounterAlign: Counterfactual Supervision for Vision-Language-Action Modelscited by 1

50 most cited of 68 All 68 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Prioritized Rollouts for Efficient World Model-based Vision-Language-Action Policy Optimization
  2. 2025SRPO: Self-Referential Policy Optimization for Vision-Language-Action Models
  3. 2026Z-1: Efficient Reinforcement Learning for Vision-Language-Action Models
  4. 2025Reinforcing Action Policies by Prophesying
  5. 2026WCM: A World Critic Model for Vision-Language-Action Reinforcement Learning
  6. 2026GigaBrain-0.5M*: a VLA That Learns From World Model-Based Reinforcement Learning
  7. 2026PAPO-VLA: Planning-Aware Policy Optimization for Vision-Language-Action Models
  8. 2026PolicyTrim: Boosting Intrinsic Policy Efficiency of Vision-Language-Action Models