# World-Gymnast: Training Robots with Reinforcement Learning in a World Model

arXiv: https://arxiv.org/abs/2602.02454 · HTML: https://mlsanity.com/paper/2602.02454/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and robot world models, one of 11 directions in World models.
- Builds on 54 of its 75 references that are in the map.
- First posted on arXiv 2 Feb 2026.

## Placed in

- World models / [Embodied and robot world models](https://mlsanity.com/wm/embodied-and-robot-world-models/)
- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 30 of 54)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [Mastering Atari with Discrete World Models](https://mlsanity.com/paper/2010.02193/) (2020)
- [Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success](https://mlsanity.com/paper/2502.19645/) (2025)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [R3M: A Universal Visual Representation for Robot Manipulation](https://mlsanity.com/paper/2203.12601/) (2022)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [BridgeData V2: A Dataset for Robot Learning at Scale](https://mlsanity.com/paper/2308.12952/) (2023)
- [Efficiently Scaling Transformer Inference](https://mlsanity.com/paper/2211.05102/) (2022)
- [Learning Universal Policies via Text-Guided Video Generation](https://mlsanity.com/paper/2302.00111/) (2023)
- [Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning](https://mlsanity.com/paper/2109.11978/) (2021)
- [Evaluating Real-World Robot Manipulation Policies in Simulation](https://mlsanity.com/paper/2405.05941/) (2024)
- [Learning Interactive Real-World Simulators](https://mlsanity.com/paper/2310.06114/) (2023)
- [Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey](https://mlsanity.com/paper/2009.13303/) (2020)
- [Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning](https://mlsanity.com/paper/2108.06266/) (2021)
- [Transformers are Sample-Efficient World Models](https://mlsanity.com/paper/2209.00588/) (2022)
- [Sim-to-Real: Learning Agile Locomotion For Quadruped Robots](https://mlsanity.com/paper/1804.10332/) (2018)
- [Deep Visual Foresight for Planning Robot Motion](https://mlsanity.com/paper/1610.00696/) (2016)
- [Any-point Trajectory Modeling for Policy Learning](https://mlsanity.com/paper/2401.00025/) (2023)
- [Ctrl-World: A Controllable Generative World Model for Robot Manipulation](https://mlsanity.com/paper/2510.10125/) (2025)
- [Learning to Act from Actionless Videos through Dense Correspondences](https://mlsanity.com/paper/2310.08576/) (2023)
- [Video Language Planning](https://mlsanity.com/paper/2310.10625/) (2023)
- [Human-to-Robot Imitation in the Wild](https://mlsanity.com/paper/2207.09450/) (2022)

## Built on by (most cited first, 27 of 27)

- [World Action Models: The Next Frontier in Embodied AI](https://mlsanity.com/paper/2605.12090/) (2026)
- [VLAW: Iterative Co-Improvement of Vision-Language-Action Policy and World Model](https://mlsanity.com/paper/2602.12063/) (2026)
- [Reconstruction or Semantics? What Makes a Latent Space Useful for Robotic World Models](https://mlsanity.com/paper/2605.06388/) (2026)
- [PlayWorld: Learning Robot World Models from Autonomous Play](https://mlsanity.com/paper/2603.09030/) (2026)
- [Code World Model: Coding Agent as World Brain](https://mlsanity.com/paper/2608.25927/) (2026)
- [Hi-WM: Human-in-the-World-Model for Scalable Robot Post-Training](https://mlsanity.com/paper/2604.21741/) (2026)
- [How Should World Models Be Evaluated for Embodied Decision-Making? A Decision-Making-Centric Position](https://mlsanity.com/paper/2606.15032/) (2026)
- [World2Act: Latent Action Post-Training from World Model Dynamics](https://mlsanity.com/paper/2603.10422/) (2026)
- [Data Pyramid for Embodied Manipulation: A Survey](https://mlsanity.com/paper/2607.24744/) (2026)
- [World Action Planner: Generalizable Decision-Making with Action-Conditioned World Models](https://mlsanity.com/paper/2607.27599/) (2026)
- [Video Generation Models as World Models: Efficient Paradigms, Architectures and Algorithms](https://mlsanity.com/paper/2603.28489/) (2026)
- [RoboWorld: Fast and Reliable Neural Simulators for Generalist Robot Policy Evaluation](https://mlsanity.com/paper/2607.01060/) (2026)
- [RigidBench: Evaluating Rigid-Body Physics in Video Generation Models](https://mlsanity.com/paper/2608.15555/) (2026)
- [Robot-Factored World Models via Robot Rendering](https://mlsanity.com/paper/2607.22535/) (2026)
- [Q-Learning With World Models](https://mlsanity.com/paper/2608.17163/) (2026)
- [Coupled Local and Global World Models for Efficient First Order RL](https://mlsanity.com/paper/2602.06219/) (2026)
- [Reinforcing VLAs in Task-Agnostic World Models](https://mlsanity.com/paper/2605.12334/) (2026)
- [From Demonstrations to Rewards: Test-Time Prompt Optimization for VLM Reward Models](https://mlsanity.com/paper/2606.00083/) (2026)
- [StressDream: Steering Video World Models for Robust Policy Evaluation and Improvement](https://mlsanity.com/paper/2606.00267/) (2026)
- [TACO: TActile World Model as a Self-COrrector for Scalable Robot Policy Post-Training](https://mlsanity.com/paper/2607.02840/) (2026)
- [PAVXploreRL: Physical-Action-Visual World Model Reinforcement Learning with Action Exploration](https://mlsanity.com/paper/2607.16602/) (2026)
- [CounterAlign: Counterfactual Supervision for Vision-Language-Action Models](https://mlsanity.com/paper/2608.21740/) (2026)
- [SyncWorld: Visual Calibration Enables World Models as Zero-Shot Simulators](https://mlsanity.com/paper/2609.09155/) (2026)
- [Pelican-Sim 1.0: A General World Model Simulator for Embodied Intelligence](https://mlsanity.com/paper/2609.12036/) (2026)
- [Toward Unified Robot Learning: Bridging Representation, Vision-Language-Action, and World Models](https://mlsanity.com/paper/2609.03927/) (2026)
- [Modeling Latent Disturbances for Robust Decision-Making in World Models](https://mlsanity.com/paper/2610.07599/) (2026)
- [Dream to Remember: Continual Steering of Generalist Robot Policies with World Models](https://mlsanity.com/paper/2610.13816/) (2026)

## Abstract (arXiv)

Robot learning from interacting with the physical world is fundamentally bottlenecked by the cost of physical interaction. The two alternatives, supervised finetuning (SFT) from expert demonstrations and reinforcement learning (RL) in a software-based simulator, are limited by the amount of expert data available and the sim-to-real gap for manipulation. With the recent emergence of world models learned from real-world video-action data, we ask the question of whether training a policy in a world model can be more effective than supervised learning or software simulation in achieving better real-robot performance. We propose World-Gymnast, which performs RL finetuning of a vision-language-action (VLA) policy by rolling out the policy in an action-conditioned video world model and rewarding the rollouts with a vision-language model (VLM). On the Bridge robot setup, World-Gymnast outperforms SFT by as much as 18x and outperforms software simulator by as much as 2x. More importantly, World-Gymnast demonstrates intriguing capabilities of RL with a world model, including training on diverse language instructions and novel scenes from the world model, test-time training in a novel scene, and online iterative world model and policy improvement. Our results suggest learning a world model and training robot policies in the cloud could be the key to bridging the gap between robots that work in demonstrations and robots that can work in anyone's household.