mlsanityData to 7 Oct 2026

Low-Variance Policy Gradient Estimation with World Models

arXiv 2010.15622 ↗First posted 29 Oct 2020stat.ML cs.AI cs.LGcited by 0

On the map: World models → Latent dynamics models and model-based RL

Abstract (arXiv)

In this paper, we propose World Model Policy Gradient (WMPG), an approach to reduce the variance of policy gradient estimates using learned world models (WM's). In WMPG, a WM is trained online and used to imagine trajectories. The imagined trajectories are used in two ways. Firstly, to calculate a without-replacement estimator of the policy gradient. Secondly, the return of the imagined trajectories is used as an informed baseline. We compare the proposed approach with AC and MAC on a set of environments of increasing complexity (CartPole, LunarLander and Pong) and find that WMPG has better sample efficiency. Based on these results, we conclude that WMPG can yield increased sample efficiency in cases where a robust latent representation of the environment can be learned.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 5 of its 38 references that are in the map. First posted on arXiv 29 Oct 2020.

Builds on

5 of 38 references.

Closest ancestors · 1

Same direction, newest first

  1. 2020Planning to Explore via Self-Supervised World Models12 May

Other references · 4

  1. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  2. 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 152
  3. 2020Incremental Sampling Without Replacement for Sequence Modelscited by 23
  4. 2020Learning Discrete State Abstractions With Deep Variational Inferencecited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023World Models via Policy-Guided Trajectory Diffusion
  2. 2026Scaling World-Model Reinforcement Learning Through Diffusion Policy Optimization
  3. 2025A Multi-Fidelity Control Variate Approach for Policy Gradient Estimation
  4. 2024PWM: Policy Learning with Multi-Task World Models
  5. 2026Q-Learning With World Models
  6. 2024Imagine-2-Drive: Leveraging High-Fidelity World Models via Multi-Modal Diffusion Policies
  7. 2025Closing the Train-Test Gap in World Models for Gradient-Based Planning
  8. 2024Towards Unraveling and Improving Generalization in World Models