mlsanityData to 7 Oct 2026

DreamerPro: Reconstruction-Free Model-Based Reinforcement Learning with Prototypical Representations

arXiv 2110.14565 ↗First posted 27 Oct 2021cs.LG cs.AIcited by 73

On the map: World models → Latent dynamics models and model-based RL

Abstract (arXiv)

Top-performing Model-Based Reinforcement Learning (MBRL) agents, such as Dreamer, learn the world model by reconstructing the image observations. Hence, they often fail to discard task-irrelevant details and struggle to handle visual distractions. To address this issue, previous work has proposed to contrastively learn the world model, but the performance tends to be inferior in the absence of distractions. In this paper, we seek to enhance robustness to distractions for MBRL agents. Specifically, we consider incorporating prototypical representations, which have yielded more accurate and robust results than contrastive approaches in computer vision. However, it remains elusive how prototypical representations can benefit temporal dynamics learning in MBRL, since they treat each image independently without capturing temporal structures. To this end, we propose to learn the prototypes from the recurrent states of the world model, thereby distilling temporal structures from past observations and actions into the prototypes. The resulting model, DreamerPro, successfully combines Dreamer with prototypes, making large performance gains on the DeepMind Control suite both in the standard setting and when there are complex background distractions. Code available at https://github.com/fdeng18/dreamer-pro .

Key facts. Cited by 73 papers in the map (data through 7 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 35 of its 41 references that are in the map. First posted on arXiv 27 Oct 2021.

Builds on

35 of 41 references.

Closest ancestors · 4

Same direction, newest first

  1. 2020Mastering Atari with Discrete World Models5 Oct
  2. 2020Dreaming: Model-based Reinforcement Learning by Latent Imagination without Reconstruction29 Jul
  3. 2019Dream to Control: Learning Behaviors by Latent Imagination3 Dec
  4. 2018Recurrent World Models Facilitate Policy Evolution4 Sep

Other references · 31

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  6. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  7. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  8. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,277
  9. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  10. 2017The Kinetics Human Action Video Datasetcited by 1,894
  11. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  12. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
19 more
  1. 2018Exploration by Random Network Distillationcited by 966
  2. 2018DeepMind Control Suitecited by 935
  3. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  4. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  5. 2019Model-Based Reinforcement Learning for Ataricited by 495
  6. 2020Reinforcement Learning with Augmented Datacited by 452
  7. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  8. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  9. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  10. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
  11. 2020Decoupling Representation Learning from Reinforcement Learningcited by 263
  12. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 204
  13. 2021Reinforcement Learning with Prototypical Representationscited by 179
  14. 2021Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learningcited by 123
  15. 2021Pretraining Representations for Data-Efficient Reinforcement Learningcited by 105
  16. 2020Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learningcited by 101
  17. 2021The Distracting Control Suite -- A Challenging Benchmark for Reinforcement Learning from Pixelscited by 97
  18. 2019Shaping Belief States with Generative Environment Models for RLcited by 65
  19. 2021Temporal Predictive Coding For Model-Based Planning In Latent Spacecited by 54

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 9; 2023: 16; 2024: 14; 2025: 14; 2026: 20.

73 papers.

Sort:

By area:

  1. 2022DayDreamer: World Models for Physical Robot Learningcited by 372
  2. 2022Masked World Models for Visual Controlcited by 169
  3. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  4. 2024World Models for Autonomous Driving: An Initial Surveycited by 79
  5. 2025A Comprehensive Survey on World Models for Embodied AIcited by 65
  6. 2024Bridging State and History Representations: Understanding Self-Predictive RLcited by 52
  7. 2025A Survey: Learning Embodied Intelligence from Physical Simulators and World Modelscited by 50
  8. 2026World Action Models: The Next Frontier in Embodied AIcited by 46
  9. 2023Pre-training Contextualized World Models with In-the-wild Videos for Reinforcement Learningcited by 45
  10. 2025Latent Action Learning Requires Supervision in the Presence of Distractorscited by 40
  11. 2023Facing Off World Model Backbones: RNNs, Transformers, and S4cited by 31
  12. 2023Hybrid Internal Model: Learning Agile Legged Locomotion with Simulated Robot Responsecited by 31
38 more
  1. 2022DreamingV2: Reinforcement Learning with Discrete World Models without Reconstructioncited by 29
  2. 2024GenRL: Multimodal-foundation world models for generalization in embodied agentscited by 29
  3. 2022Simplifying Model-based RL: Learning Representations, Latent-space Models, and Policies with One Objectivecited by 28
  4. 2025GEVRM: Goal-Expressive Video Generation Model For Robust Visual Manipulationcited by 26
  5. 2023Learning World Models with Identifiable Factorizationcited by 23
  6. 2023HarmonyDream: Task Harmonization Inside World Modelscited by 22
  7. 2022Mastering the Unsupervised Reinforcement Learning Benchmark from Pixelscited by 20
  8. 2024Learning Latent Dynamic Robust Representations for World Modelscited by 20
  9. 2025Learning Transformer-based World Models with Contrastive Predictive Codingcited by 20
  10. 2023FOCUS: Object-Centric World Models for Robotics Manipulationcited by 17
  11. 2023Curious Replay for Model-based Adaptationcited by 16
  12. 2026Predictive but Not Plannable: RC-aux for Latent World Modelscited by 16
  13. 2023Robust Representation Learning by Clustering with Bisimulation Metrics for Visual Reinforcement Learning with Distractionscited by 13
  14. 2024MuDreamer: Learning Predictive World Models without Reconstructioncited by 13
  15. 2026R2-Dreamer: Redundancy-Reduced World Models without Decoders or Augmentationcited by 10
  16. 2022S2P: State-conditioned Image Synthesis for Data Augmentation in Offline Reinforcement Learningcited by 9
  17. 2023Combining Reconstruction and Contrastive Methods for Multimodal Representations in RLcited by 9
  18. 2023Cross-domain Random Pre-training with Prototypes for Reinforcement Learningcited by 9
  19. 2022Learning Robust Dynamics through Variational Sparse Gatingcited by 8
  20. 2023Learning Better with Less: Effective Augmentation for Sample-Efficient Visual Reinforcement Learningcited by 8
  21. 2023DreamSmooth: Improving Model-based Reinforcement Learning via Reward Smoothingcited by 8
  22. 2025Reinforcement Fine-Tuning of Flow-Matching Policies for Vision-Language-Action Modelscited by 7
  23. 2024Policy-shaped prediction: avoiding distractions in model-based reinforcement learningcited by 6
  24. 2023Task Aware Dreamer for Task Generalization in Reinforcement Learningcited by 5
  25. 2026Aligning Agentic World Models via Knowledgeable Experience Learningcited by 5
  26. 2023Model-Based Reinforcement Learning with Isolated Imaginationscited by 4
  27. 2024Make the Pertinent Salient: Task-Relevant Reconstruction for Visual Control with Distractionscited by 4
  28. 2024Sample-efficient Unsupervised Policy Cloning from Ensemble Self-supervised Labeled Videoscited by 4
  29. 2025Videos are Sample-Efficient Supervisions: Behavior Cloning from Videos via Latent Representationscited by 4
  30. 2026Next Embedding Prediction Makes World Models Strongercited by 4
  31. 2026Temporally Centered SIGReg Improves LeWorldModel Representations for Robot Policy Learningcited by 4
  32. 2022Prototypical context-aware dynamics generalization for high-dimensional model-based reinforcement learningcited by 3
  33. 2022Cost-Driven Representation Learning for Linear Quadratic Gaussian Control: Part Icited by 3
  34. 2023Sample-efficient Real-time Planning with Curiosity Cross-Entropy Method and Contrastive Learningcited by 3
  35. 2023Iteratively Learn Diverse Strategies with State Distance Informationcited by 3
  36. 2024Episodic Reinforcement Learning with Expanded State-reward Spacecited by 3
  37. 2024Learning Future Representation with Synthetic Observations for Sample-efficient Reinforcement Learningcited by 3
  38. 2025TAR: Teacher-Aligned Representations via Contrastive Learning for Quadrupedal Locomotioncited by 3

50 most cited of 73 All 73 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Zero-shot World Models via Search in Memory
  2. 2026Reinformed Dreamer: An Asymmetric World Model Efficiently Trained through Latent Guidance
  3. 2022Iso-Dream: Isolating and Leveraging Noncontrollable Visual Dynamics in World Models
  4. 2018Learning Real-World Robot Policies by Dreaming
  5. 2025Multimodal Dreaming: A Global Workspace Approach to World Model-Based Reinforcement Learning
  6. 2022Multi-View Dreaming: Multi-View World Model with Contrastive Learning
  7. 2024CURLing the Dream: Contrastive Representations for World Modeling in Reinforcement Learning
  8. 2024Dreaming of Many Worlds: Learning Contextual World Models Aids Zero-Shot Generalization