# RLVR-World: Training World Models with Reinforcement Learning

arXiv: https://arxiv.org/abs/2505.13934 · HTML: https://mlsanity.com/paper/2505.13934/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Placed in Latent dynamics models and model-based RL, one of 11 directions in World models.
- Builds on 67 of its 81 references that are in the map.
- First posted on arXiv 20 May 2025.

## Placed in

- World models / [Latent dynamics models and model-based RL](https://mlsanity.com/wm/latent-dynamics-models-and-model-based-rl/)

## Builds on (most cited first, 30 of 67)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Scaling Laws for Reward Model Overoptimization](https://mlsanity.com/paper/2210.10760/) (2022)
- [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://mlsanity.com/paper/2411.15124/) (2024)
- [ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2304.05977/) (2023)
- [Training Diffusion Models with Reinforcement Learning](https://mlsanity.com/paper/2305.13301/) (2023)

## Built on by (most cited first, 30 of 40)

- [A Comprehensive Survey on World Models for Embodied AI](https://mlsanity.com/paper/2510.16732/) (2025)
- [From Word to World: Can Large Language Models be Implicit Text-based World Models?](https://mlsanity.com/paper/2512.18832/) (2025)
- [Planning with Reasoning using Vision Language World Model](https://mlsanity.com/paper/2509.02722/) (2025)
- [Qwen-AgentWorld: Language World Models for General Agents](https://mlsanity.com/paper/2606.24597/) (2026)
- [What about gravity in video generation? Post-Training Newton's Laws with Verifiable Rewards](https://mlsanity.com/paper/2512.00425/) (2025)
- [Visual Generation Unlocks Human-Like Reasoning through Multimodal World Models](https://mlsanity.com/paper/2601.19834/) (2026)
- [Can One Domain Help Others? A Data-Centric Study on Multi-Domain Reasoning via Reinforcement Learning](https://mlsanity.com/paper/2507.17512/) (2025)
- [MobileVLA-R1: Reinforcing Vision-Language-Action for Mobile Robots](https://mlsanity.com/paper/2511.17889/) (2025)
- [EVA: Aligning Video World Models with Executable Robot Actions via Inverse Dynamics Rewards](https://mlsanity.com/paper/2603.17808/) (2026)
- [Reinforcement Learning with Inverse Rewards for World Model Post-training](https://mlsanity.com/paper/2509.23958/) (2025)
- [Integrating Reinforcement Learning with Visual Generative Models: Foundations and Advances](https://mlsanity.com/paper/2508.10316/) (2025)
- [Persistent Robot World Models: Stabilizing Multi-Step Rollouts via Reinforcement Learning](https://mlsanity.com/paper/2603.25685/) (2026)
- [Is the Future Compatible? Diagnosing Dynamic Consistency in World Action Models](https://mlsanity.com/paper/2605.07514/) (2026)
- [RoboAlign-R1: Distilled Multimodal Reward Alignment for Robot Video World Models](https://mlsanity.com/paper/2605.03821/) (2026)
- [RAVEN: Real-time Autoregressive Video Extrapolation with Consistency-model GRPO](https://mlsanity.com/paper/2605.15190/) (2026)
- [Towards High-Consistency Embodied World Model with Multi-View Trajectory Videos](https://mlsanity.com/paper/2511.12882/) (2025)
- [AgentWatcher: A Rule-based Prompt Injection Monitor](https://mlsanity.com/paper/2604.01194/) (2026)
- [Overcoming Statistical Bias in Action-Controllable World Models](https://mlsanity.com/paper/2608.04653/) (2026)
- [Beyond State Consistency: Behavior Consistency in Text-Based World Models](https://mlsanity.com/paper/2604.13824/) (2026)
- [Low-rank Optimization Trajectories Modeling for LLM RLVR Acceleration](https://mlsanity.com/paper/2604.11446/) (2026)
- [TRAP: Tail-aware Ranking Attack for World-Model Planning](https://mlsanity.com/paper/2605.01950/) (2026)
- [Learning Visual Feature-Based World Models via Residual Latent Action](https://mlsanity.com/paper/2605.07079/) (2026)
- [Twin Rollouts: Noise-Coupled Counterfactual Branching in Interactive Video World Models](https://mlsanity.com/paper/2608.08982/) (2026)
- [OpenWorldLib: A Unified Codebase and Definition of Advanced World Models](https://mlsanity.com/paper/2604.04707/) (2026)
- [World Models: A Comprehensive Survey of Architectures, Methodologies, Reasoning Paradigms, and Applications](https://mlsanity.com/paper/2606.00133/) (2026)
- [WorldCycle: Self-Verifiable Reinforcement Learning for Long-Horizon Video World Models](https://mlsanity.com/paper/2608.04964/) (2026)
- [Agentic Game Development as a Verifiable Trajectory Data Engine for Scaling World Models](https://mlsanity.com/paper/2608.25518/) (2026)
- [ARTIS: Agentic Risk-Aware Test-Time Scaling via Iterative Simulation](https://mlsanity.com/paper/2602.01709/) (2026)
- [LaViPlan : Language-Guided Visual Path Planning with RLVR](https://mlsanity.com/paper/2507.12911/) (2025)
- [ScRPO: From Errors to Insights](https://mlsanity.com/paper/2511.06065/) (2025)

## Abstract (arXiv)

World models predict state transitions in response to actions and are increasingly developed across diverse modalities. However, standard training objectives such as maximum likelihood estimation (MLE) often misalign with task-specific goals of world models, i.e., transition prediction metrics like accuracy or perceptual quality. In this paper, we present RLVR-World, a unified framework that leverages reinforcement learning with verifiable rewards (RLVR) to directly optimize world models for such metrics. Despite formulating world modeling as autoregressive prediction of tokenized sequences, RLVR-World evaluates metrics of decoded predictions as verifiable rewards. We demonstrate substantial performance gains on both language- and video-based world models across domains, including text games, web navigation, and robot manipulation. Our work indicates that, beyond recent advances in reasoning language models, RLVR offers a promising post-training paradigm for enhancing the utility of generative models more broadly. Code, datasets, models, and video samples are available at the project website: https://thuml.github.io/RLVR-World.