# Reinforcing Action Policies by Prophesying

arXiv: https://arxiv.org/abs/2511.20633 · HTML: https://mlsanity.com/paper/2511.20633/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and robot world models, one of 11 directions in World models.
- Builds on 0 of its 76 references that are in the map.
- First posted on arXiv 25 Nov 2025.

## Placed in

- World models / [Embodied and robot world models](https://mlsanity.com/wm/embodied-and-robot-world-models/)
- Vision-language-action models / [Reinforcement learning and post-training](https://mlsanity.com/vla/reinforcement-learning-and-post-training/)

## Built on by (most cited first, 22 of 22)

- [DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos](https://mlsanity.com/paper/2602.06949/) (2026)
- [World Model for Robot Learning: A Comprehensive Survey](https://mlsanity.com/paper/2605.00080/) (2026)
- [RISE: Self-Improving Robot Policy with Compositional World Model](https://mlsanity.com/paper/2602.11075/) (2026)
- [World-VLA-Loop: Closed-Loop Learning of Video World Model and VLA Policy](https://mlsanity.com/paper/2602.06508/) (2026)
- [Reconstruction or Semantics? What Makes a Latent Space Useful for Robotic World Models](https://mlsanity.com/paper/2605.06388/) (2026)
- [Towards Long-Lived Robots: Continual Learning VLA Models via Reinforcement Fine-Tuning](https://mlsanity.com/paper/2602.10503/) (2026)
- [Motus2: A Self-Evolving General World Model for Dexterous Manipulation](https://mlsanity.com/paper/2608.30237/) (2026)
- [Learning Transferable Dynamics Priors from Action to World Modeling](https://mlsanity.com/paper/2606.29501/) (2026)
- [Data Pyramid for Embodied Manipulation: A Survey](https://mlsanity.com/paper/2607.24744/) (2026)
- [IG-RFT: An Interaction-Guided RL Framework for VLA Models in Long-Horizon Robotic Manipulation](https://mlsanity.com/paper/2602.20715/) (2026)
- [WorldSample: Closed-loop Real-robot RL with World Modelling](https://mlsanity.com/paper/2607.02431/) (2026)
- [Policy Mirror Descent with Temporal Difference Learning: Sample Complexity under Online Markov Data](https://mlsanity.com/paper/2512.24056/) (2025)
- [Dual-Granularity Contrastive Reward via Generated Episodic Guidance for Efficient Embodied RL](https://mlsanity.com/paper/2602.12636/) (2026)
- [Advances in GRPO for Generation Models: A Survey](https://mlsanity.com/paper/2603.06623/) (2026)
- [DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation](https://mlsanity.com/paper/2607.13033/) (2026)
- [Reinforcing VLAs in Task-Agnostic World Models](https://mlsanity.com/paper/2605.12334/) (2026)
- [PAVXploreRL: Physical-Action-Visual World Model Reinforcement Learning with Action Exploration](https://mlsanity.com/paper/2607.16602/) (2026)
- [World Models for Embodied Intelligence: From Plausible to Controllable to Actionable](https://mlsanity.com/paper/2609.16697/) (2026)
- [Scaling by Diversified Experience for Vision-Language-Action Models](https://mlsanity.com/paper/2606.09009/) (2026)
- [ROVE: Unlocking Human Interventions for Humanoid Manipulation via Reinforcement Learning](https://mlsanity.com/paper/2606.17011/) (2026)
- [Uncertainty-Gated Exploration Noise Suppresses Task Collapse in Online RL Fine-Tuning of a Flow-Matching Vision-Language-Action Policy](https://mlsanity.com/paper/2609.28838/) (2026)
- [DriftOPD: Sequence-Level Reverse-KL Distillation for One-Step VLA Policies](https://mlsanity.com/paper/2610.00317/) (2026)

## Abstract (arXiv)

Vision-Language-Action (VLA) policies excel in aligning language, perception, and robot control. However, most VLAs are trained purely by imitation, which overfits to demonstrations, and is brittle under distribution shift. Reinforcement learning (RL) directly optimizes task reward and thus addresses this misalignment, but real-robot interaction is expensive and conventional simulators are hard to engineer and transfer. We address both data efficiency and optimization stability in VLA post-training via a learned world model and an RL procedure tailored to flow-based action heads. Specifically, we first introduce Prophet, a unified action-to-video robot world model pretrained on large-scale, heterogeneous robot data to learn reusable action-outcome dynamics and then few-shot adapted to new robots, objects, and environments, yielding a rollout-ready simulator. Upon Prophet, we reinforce action policies with our proposed FlowScale, which couples Flow-GRPO with intrinsic stepwise reweighting to stabilize gradients. Together, our solution provides a practical, data- and compute-efficient path to VLA post-training. Experiments show 5-17% success gains on public benchmarks and 24-30% on real robots across diverse VLA backbones.