# DreamerPro: Reconstruction-Free Model-Based Reinforcement Learning with Prototypical Representations

arXiv: https://arxiv.org/abs/2110.14565 · HTML: https://mlsanity.com/paper/2110.14565/

## Key facts

- Cited by 74 papers in the map (data through 14 Oct 2026).
- Placed in Latent dynamics models and model-based RL, one of 11 directions in World models.
- Builds on 35 of its 41 references that are in the map.
- First posted on arXiv 27 Oct 2021.

## Placed in

- World models / [Latent dynamics models and model-based RL](https://mlsanity.com/wm/latent-dynamics-models-and-model-based-rl/)

## Builds on (most cited first, 30 of 35)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Mastering Atari with Discrete World Models](https://mlsanity.com/paper/2010.02193/) (2020)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [DeepMind Control Suite](https://mlsanity.com/paper/1801.00690/) (2018)
- [Recurrent World Models Facilitate Policy Evolution](https://mlsanity.com/paper/1809.01999/) (2018)
- [CURL: Contrastive Unsupervised Representations for Reinforcement Learning](https://mlsanity.com/paper/2004.04136/) (2020)
- [Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels](https://mlsanity.com/paper/2004.13649/) (2020)
- [Model-Based Reinforcement Learning for Atari](https://mlsanity.com/paper/1903.00374/) (2019)
- [Reinforcement Learning with Augmented Data](https://mlsanity.com/paper/2004.14990/) (2020)
- [Learning Invariant Representations for Reinforcement Learning without Reconstruction](https://mlsanity.com/paper/2006.10742/) (2020)
- [Reinforcement Learning with Unsupervised Auxiliary Tasks](https://mlsanity.com/paper/1611.05397/) (2016)
- [Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning](https://mlsanity.com/paper/2107.09645/) (2021)
- [Data-Efficient Reinforcement Learning with Self-Predictive Representations](https://mlsanity.com/paper/2007.05929/) (2020)
- [Decoupling Representation Learning from Reinforcement Learning](https://mlsanity.com/paper/2009.08319/) (2020)
- [DeepMDP: Learning Continuous Latent Space Models for Representation Learning](https://mlsanity.com/paper/1906.02736/) (2019)
- [Reinforcement Learning with Prototypical Representations](https://mlsanity.com/paper/2102.11271/) (2021)
- [Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning](https://mlsanity.com/paper/2101.05265/) (2021)
- [Pretraining Representations for Data-Efficient Reinforcement Learning](https://mlsanity.com/paper/2106.04799/) (2021)

## Built on by (most cited first, 30 of 74)

- [DayDreamer: World Models for Physical Robot Learning](https://mlsanity.com/paper/2206.14176/) (2022)
- [Masked World Models for Visual Control](https://mlsanity.com/paper/2206.14244/) (2022)
- [Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyond](https://mlsanity.com/paper/2405.03520/) (2024)
- [World Models for Autonomous Driving: An Initial Survey](https://mlsanity.com/paper/2403.02622/) (2024)
- [A Comprehensive Survey on World Models for Embodied AI](https://mlsanity.com/paper/2510.16732/) (2025)
- [Bridging State and History Representations: Understanding Self-Predictive RL](https://mlsanity.com/paper/2401.08898/) (2024)
- [A Survey: Learning Embodied Intelligence from Physical Simulators and World Models](https://mlsanity.com/paper/2507.00917/) (2025)
- [World Action Models: The Next Frontier in Embodied AI](https://mlsanity.com/paper/2605.12090/) (2026)
- [Pre-training Contextualized World Models with In-the-wild Videos for Reinforcement Learning](https://mlsanity.com/paper/2305.18499/) (2023)
- [Latent Action Learning Requires Supervision in the Presence of Distractors](https://mlsanity.com/paper/2502.00379/) (2025)
- [Hybrid Internal Model: Learning Agile Legged Locomotion with Simulated Robot Response](https://mlsanity.com/paper/2312.11460/) (2023)
- [Facing Off World Model Backbones: RNNs, Transformers, and S4](https://mlsanity.com/paper/2307.02064/) (2023)
- [GenRL: Multimodal-foundation world models for generalization in embodied agents](https://mlsanity.com/paper/2406.18043/) (2024)
- [DreamingV2: Reinforcement Learning with Discrete World Models without Reconstruction](https://mlsanity.com/paper/2203.00494/) (2022)
- [Simplifying Model-based RL: Learning Representations, Latent-space Models, and Policies with One Objective](https://mlsanity.com/paper/2209.08466/) (2022)
- [GEVRM: Goal-Expressive Video Generation Model For Robust Visual Manipulation](https://mlsanity.com/paper/2502.09268/) (2025)
- [Learning World Models with Identifiable Factorization](https://mlsanity.com/paper/2306.06561/) (2023)
- [HarmonyDream: Task Harmonization Inside World Models](https://mlsanity.com/paper/2310.00344/) (2023)
- [Mastering the Unsupervised Reinforcement Learning Benchmark from Pixels](https://mlsanity.com/paper/2209.12016/) (2022)
- [Learning Latent Dynamic Robust Representations for World Models](https://mlsanity.com/paper/2405.06263/) (2024)
- [Learning Transformer-based World Models with Contrastive Predictive Coding](https://mlsanity.com/paper/2503.04416/) (2025)
- [Predictive but Not Plannable: RC-aux for Latent World Models](https://mlsanity.com/paper/2605.07278/) (2026)
- [FOCUS: Object-Centric World Models for Robotics Manipulation](https://mlsanity.com/paper/2307.02427/) (2023)
- [Curious Replay for Model-based Adaptation](https://mlsanity.com/paper/2306.15934/) (2023)
- [Robust Representation Learning by Clustering with Bisimulation Metrics for Visual Reinforcement Learning with Distractions](https://mlsanity.com/paper/2302.12003/) (2023)
- [MuDreamer: Learning Predictive World Models without Reconstruction](https://mlsanity.com/paper/2405.15083/) (2024)
- [R2-Dreamer: Redundancy-Reduced World Models without Decoders or Augmentation](https://mlsanity.com/paper/2603.18202/) (2026)
- [S2P: State-conditioned Image Synthesis for Data Augmentation in Offline Reinforcement Learning](https://mlsanity.com/paper/2209.15256/) (2022)
- [Combining Reconstruction and Contrastive Methods for Multimodal Representations in RL](https://mlsanity.com/paper/2302.05342/) (2023)
- [Cross-domain Random Pre-training with Prototypes for Reinforcement Learning](https://mlsanity.com/paper/2302.05614/) (2023)

## Abstract (arXiv)

Top-performing Model-Based Reinforcement Learning (MBRL) agents, such as Dreamer, learn the world model by reconstructing the image observations. Hence, they often fail to discard task-irrelevant details and struggle to handle visual distractions. To address this issue, previous work has proposed to contrastively learn the world model, but the performance tends to be inferior in the absence of distractions. In this paper, we seek to enhance robustness to distractions for MBRL agents. Specifically, we consider incorporating prototypical representations, which have yielded more accurate and robust results than contrastive approaches in computer vision. However, it remains elusive how prototypical representations can benefit temporal dynamics learning in MBRL, since they treat each image independently without capturing temporal structures. To this end, we propose to learn the prototypes from the recurrent states of the world model, thereby distilling temporal structures from past observations and actions into the prototypes. The resulting model, DreamerPro, successfully combines Dreamer with prototypes, making large performance gains on the DeepMind Control suite both in the standard setting and when there are complex background distractions. Code available at https://github.com/fdeng18/dreamer-pro .