# WorldSample: Closed-loop Real-robot RL with World Modelling

arXiv: https://arxiv.org/abs/2607.02431 · HTML: https://mlsanity.com/paper/2607.02431/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and robot world models, one of 11 directions in World models.
- Builds on 28 of its 33 references that are in the map.
- First posted on arXiv 2 Jul 2026.

## Placed in

- World models / [Embodied and robot world models](https://mlsanity.com/wm/embodied-and-robot-world-models/)
- Offline RL and imitation learning from data / [Robot manipulation imitation learning](https://mlsanity.com/offlinerl/robot-manipulation-imitation-learning/)
- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 28 of 28)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025)
- [RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation](https://mlsanity.com/paper/2410.07864/) (2024)
- [MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations](https://mlsanity.com/paper/2310.17596/) (2023)
- [Diffusion Policy Policy Optimization](https://mlsanity.com/paper/2409.00588/) (2024)
- [VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning](https://mlsanity.com/paper/2505.18719/) (2025)
- [Steering Your Diffusion Policy with Latent Space Reinforcement Learning](https://mlsanity.com/paper/2506.15799/) (2025)
- [ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policy](https://mlsanity.com/paper/2502.05450/) (2025)
- [SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning](https://mlsanity.com/paper/2509.09674/) (2025)
- [NORA: A Small Open-Sourced Generalist Vision Language Action Model for Embodied Tasks](https://mlsanity.com/paper/2504.19854/) (2025)
- [Reinforcement Learning for Long-Horizon Interactive LLM Agents](https://mlsanity.com/paper/2502.01600/) (2025)
- [WMPO: World Model-based Policy Optimization for Vision-Language-Action Models](https://mlsanity.com/paper/2511.09515/) (2025)
- [DemoGen: Synthetic Demonstration Generation for Data-Efficient Visuomotor Policy Learning](https://mlsanity.com/paper/2502.16932/) (2025)
- [VLA-RFT: Vision-Language-Action Reinforcement Fine-tuning with Verified Rewards in World Simulators](https://mlsanity.com/paper/2510.00406/) (2025)
- [World-Env: Leveraging World Model as a Virtual Environment for VLA Post-Training](https://mlsanity.com/paper/2509.24948/) (2025)
- [WoVR: World Models as Reliable Simulators for Post-Training VLA Policies with RL](https://mlsanity.com/paper/2602.13977/) (2026)
- [ReinboT: Amplifying Robot Visual-Language Manipulation with Reinforcement Learning](https://mlsanity.com/paper/2505.07395/) (2025)
- [VLAW: Iterative Co-Improvement of Vision-Language-Action Policy and World Model](https://mlsanity.com/paper/2602.12063/) (2026)
- [RISE: Self-Improving Robot Policy with Compositional World Model](https://mlsanity.com/paper/2602.11075/) (2026)
- [CO-RFT: Efficient Fine-Tuning of Vision-Language-Action Models through Chunked Offline Reinforcement Learning](https://mlsanity.com/paper/2508.02219/) (2025)
- [World-VLA-Loop: Closed-Loop Learning of Video World Model and VLA Policy](https://mlsanity.com/paper/2602.06508/) (2026)
- [Self-Improving Embodied Foundation Models](https://mlsanity.com/paper/2509.15155/) (2025)
- [RLinf-VLA: A Unified and Efficient Framework for Reinforcement Learning of Vision-Language-Action Models](https://mlsanity.com/paper/2510.06710/) (2025)
- [Reinforcing Action Policies by Prophesying](https://mlsanity.com/paper/2511.20633/) (2025)
- [SIME: Enhancing Policy Self-Improvement with Modal-level Exploration](https://mlsanity.com/paper/2505.01396/) (2025)
- [RESample: A Robust Data Augmentation Framework via Exploratory Sampling for Robotic Manipulation](https://mlsanity.com/paper/2510.17640/) (2025)

## Built on by (most cited first, 3 of 3)

- [WAM-OPD: On-Policy Distillation for World Action Models](https://mlsanity.com/paper/2608.22364/) (2026)
- [World Models for Embodied Intelligence: From Plausible to Controllable to Actionable](https://mlsanity.com/paper/2609.16697/) (2026)
- [UniIntervene++: An Adaptive Intervention Agent for Efficient Real-World Reinforcement Learning](https://mlsanity.com/paper/2610.03620/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) can overcome the demonstration-coverage limitation of imitation learning (IL) by allowing robots to improve through trial-and-error interaction beyond the states observed in demonstrations. However, deploying RL on real robots remains constrained by high interaction costs, since each physical rollout is costly and reflects only one realized action-outcome path. To address this challenge, we propose WorldSample, a physically grounded data augmentation framework for real-robot RL that closes a real-synthetic loop between physical rollouts, world-model generation, and policy improvement. Grounded on real rollouts, WorldSample generates high-fidelity synthetic transitions through a post-trained world model, which greatly lowers the visual hallucination. Specifically, rather than simply using these transitions as real-world experience, WorldSample introduces Policy-Paced Learning (PPL) to regulate the training process through sample selection and scheduling, balancing useful augmentation against value overestimation and mitigating the hallucination-induced noise. Experiments on robot manipulation tasks involving contact-rich and precise tasks show that WorldSample improves policy success rate by 28% while reducing training steps by 59% compared with baselines. Furthermore, WorldSample improves world model visual fidelity by 19.4dB in PSNR and 0.47 in SSIM over demonstration-only post-training, validating the effectiveness of the real-synthetic loop for both policy and world model performance.