# Persistent Robot World Models: Stabilizing Multi-Step Rollouts via Reinforcement Learning

arXiv: https://arxiv.org/abs/2603.25685 · HTML: https://mlsanity.com/paper/2603.25685/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and robot world models, one of 11 directions in World models.
- Builds on 44 of its 55 references that are in the map.
- First posted on arXiv 26 Mar 2026.

## Placed in

- World models / [Embodied and robot world models](https://mlsanity.com/wm/embodied-and-robot-world-models/)
- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 30 of 44)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [GR00T N1: An Open Foundation Model for Generalist Humanoid Robots](https://mlsanity.com/paper/2503.14734/) (2025)
- [Training Diffusion Models with Reinforcement Learning](https://mlsanity.com/paper/2305.13301/) (2023)
- [Open X-Embodiment: Robotic Learning Datasets and RT-X Models](https://mlsanity.com/paper/2310.08864/) (2023)
- [DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset](https://mlsanity.com/paper/2403.12945/) (2024)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [Diffusion Model Alignment Using Direct Preference Optimization](https://mlsanity.com/paper/2311.12908/) (2023)
- [FAST: Efficient Action Tokenization for Vision-Language-Action Models](https://mlsanity.com/paper/2501.09747/) (2025)
- [Flow-GRPO: Training Flow Matching Models via Online RL](https://mlsanity.com/paper/2505.05470/) (2025)
- [Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion](https://mlsanity.com/paper/2506.08009/) (2025)
- [Sequence Level Training with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06732/) (2015)
- [Learning Universal Policies via Text-Guided Video Generation](https://mlsanity.com/paper/2302.00111/) (2023)
- [Evaluating Real-World Robot Manipulation Policies in Simulation](https://mlsanity.com/paper/2405.05941/) (2024)
- [DanceGRPO: Unleashing GRPO on Visual Generation](https://mlsanity.com/paper/2505.07818/) (2025)
- [Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations](https://mlsanity.com/paper/2412.14803/) (2024)
- [Diffusion Policy Policy Optimization](https://mlsanity.com/paper/2409.00588/) (2024)
- [Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models](https://mlsanity.com/paper/2310.10639/) (2023)

## Built on by (most cited first, 12 of 12)

- [ImageWAM: Do World Action Models Really Need Video Generation, or Just Image Editing?](https://mlsanity.com/paper/2606.19531/) (2026)
- [How Should World Models Be Evaluated for Embodied Decision-Making? A Decision-Making-Centric Position](https://mlsanity.com/paper/2606.15032/) (2026)
- [World-Action Models for Robot Learning and Control: A Survey](https://mlsanity.com/paper/2609.16074/) (2026)
- [Spatially Aware World Action Model via Geometric Latent Diffusion](https://mlsanity.com/paper/2609.02531/) (2026)
- [RoboWorld: Fast and Reliable Neural Simulators for Generalist Robot Policy Evaluation](https://mlsanity.com/paper/2607.01060/) (2026)
- [Twin Rollouts: Noise-Coupled Counterfactual Branching in Interactive Video World Models](https://mlsanity.com/paper/2608.08982/) (2026)
- [PAVXploreRL: Physical-Action-Visual World Model Reinforcement Learning with Action Exploration](https://mlsanity.com/paper/2607.16602/) (2026)
- [XPACE: Joint World and Action Modeling from Heterogeneous Experience](https://mlsanity.com/paper/2609.17372/) (2026)
- [Learning to Act under Visual Interruptions with Vision-Language-Action Models](https://mlsanity.com/paper/2609.35003/) (2026)
- [EVEWorld: Physical Evolution Supervision for Embodied World Models](https://mlsanity.com/paper/2610.03374/) (2026)
- [SimForcing: Distilling Simulation Motion Priors into Real-Domain Robot World Models](https://mlsanity.com/paper/2610.06598/) (2026)
- [DepthWorld: 3D World Model for Robot Manipulation](https://mlsanity.com/paper/2610.08780/) (2026)

## Abstract (arXiv)

Action-conditioned robot world models generate future video frames of the manipulated scene given a robot action sequence, offering a promising alternative for simulating tasks that are difficult to model with traditional physics engines. However, these models are optimized for short-term prediction and break down when deployed autoregressively: each predicted clip feeds back as context for the next, causing errors to compound and visual quality to rapidly degrade. We address this through the following contributions. First, we introduce a reinforcement learning (RL) post-training scheme that trains the world model on its own autoregressive rollouts rather than on ground-truth histories. We achieve this by adapting a recent contrastive RL objective for diffusion models to our setting and show that its convergence guarantees carry over exactly. Second, we design a training protocol that generates and compares multiple candidate variable-length futures from the same rollout state, reinforcing higher-fidelity predictions over lower-fidelity ones. Third, we develop efficient, multi-view visual fidelity rewards that combine complementary perceptual metrics across camera views and are aggregated at the clip level for dense, low-variance training signal. Fourth, we show that our approach establishes a new state-of-the-art for rollout fidelity on the DROID dataset, outperforming the strongest baseline on all metrics (e.g., LPIPS reduced by 14% on external cameras, SSIM improved by 9.1% on the wrist camera), winning 98% of paired comparisons, and achieving an 80% preference rate in a blind human study.