# Coupled Local and Global World Models for Efficient First Order RL

arXiv: https://arxiv.org/abs/2602.06219 · HTML: https://mlsanity.com/paper/2602.06219/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and robot world models, one of 11 directions in World models.
- Builds on 38 of its 43 references that are in the map.
- First posted on arXiv 5 Feb 2026.

## Placed in

- World models / [Embodied and robot world models](https://mlsanity.com/wm/embodied-and-robot-world-models/)

## Builds on (most cited first, 30 of 38)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Mastering Atari with Discrete World Models](https://mlsanity.com/paper/2010.02193/) (2020)
- [Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models](https://mlsanity.com/paper/1805.12114/) (2018)
- [When to Trust Your Model: Model-Based Policy Optimization](https://mlsanity.com/paper/1906.08253/) (2019)
- [Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion](https://mlsanity.com/paper/2407.01392/) (2024)
- [TD-MPC2: Scalable, Robust World Models for Continuous Control](https://mlsanity.com/paper/2310.16828/) (2023)
- [Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey](https://mlsanity.com/paper/2009.13303/) (2020)
- [DayDreamer: World Models for Physical Robot Learning](https://mlsanity.com/paper/2206.14176/) (2022)
- [Temporal Difference Learning for Model Predictive Control](https://mlsanity.com/paper/2203.04955/) (2022)
- [One Step Diffusion via Shortcut Models](https://mlsanity.com/paper/2410.12557/) (2024)
- [Axial Attention in Multidimensional Transformers](https://mlsanity.com/paper/1912.12180/) (2019)
- [Diffusion for World Modeling: Visual Details Matter in Atari](https://mlsanity.com/paper/2405.12399/) (2024)
- [Model-Ensemble Trust-Region Policy Optimization](https://mlsanity.com/paper/1802.10592/) (2018)
- [Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion](https://mlsanity.com/paper/1807.01675/) (2018)
- [Extreme Parkour with Legged Robots](https://mlsanity.com/paper/2309.14341/) (2023)
- [ANYmal Parkour: Learning Agile Navigation for Quadrupedal Robots](https://mlsanity.com/paper/2306.14874/) (2023)
- [Robot Parkour Learning](https://mlsanity.com/paper/2309.05665/) (2023)
- [Training Agents Inside of Scalable World Models](https://mlsanity.com/paper/2509.24527/) (2025)
- [SONIC: Supersizing Motion Tracking for Natural Humanoid Whole-Body Control](https://mlsanity.com/paper/2511.07820/) (2025)
- [Model-Augmented Actor-Critic: Backpropagating through Paths](https://mlsanity.com/paper/2005.08068/) (2020)
- [Investigating Compounding Prediction Errors in Learned Dynamics Models](https://mlsanity.com/paper/2203.09637/) (2022)
- [Learning to Combat Compounding-Error in Model-Based Reinforcement Learning](https://mlsanity.com/paper/1912.11206/) (2019)
- [Rank2Reward: Learning Shaped Reward Functions from Passive Video](https://mlsanity.com/paper/2404.14735/) (2024)
- [World-Gymnast: Training Robots with Reinforcement Learning in a World Model](https://mlsanity.com/paper/2602.02454/) (2026)
- [CaT: Constraints as Terminations for Legged Locomotion Reinforcement Learning](https://mlsanity.com/paper/2403.18765/) (2024)

## Built on by (most cited first, 1 of 1)

- [FLEX-WAM: Flexible Block-Causal World-Action Models for Long-Horizon Imagination and Planning](https://mlsanity.com/paper/2610.05483/) (2026)

## Abstract (arXiv)

World models offer a promising avenue for more faithfully capturing complex dynamics, including contacts and non-rigidity, as well as complex sensory information, such as visual perception, in situations where standard simulators struggle. However, these models are computationally complex to evaluate, posing a challenge for popular RL approaches that have been successfully used with simulators to solve complex locomotion tasks but yet struggle with manipulation. This paper introduces a method that bypasses simulators entirely, training RL policies inside world models learned from robots' interactions with real environments. At its core, our approach enables policy training with large-scale diffusion models via a novel decoupled first-order gradient (FoG) method: a full-scale world model generates accurate forward trajectories, while a lightweight latent-space surrogate approximates its local dynamics for efficient gradient computation. This coupling of a local and global world model ensures high-fidelity unrolling alongside computationally tractable differentiation. We demonstrate the efficacy of our method on the Push-T manipulation task, where it significantly outperforms PPO in sample efficiency. We further evaluate our approach through an ego-centric object manipulation task with a quadruped. Together, these results demonstrate that learning inside data-driven world models is a promising pathway for solving hard-to-model RL tasks in image space without reliance on hand-crafted physics simulators.