# Denoised MDPs: Learning World Models Better Than the World Itself

arXiv: https://arxiv.org/abs/2206.15477 · HTML: https://mlsanity.com/paper/2206.15477/

## Key facts

- Cited by 68 papers in the map (data through 14 Oct 2026).
- Placed in Latent dynamics models and model-based RL, one of 11 directions in World models.
- Builds on 35 of its 55 references that are in the map.
- First posted on arXiv 30 Jun 2022.

## Placed in

- World models / [Latent dynamics models and model-based RL](https://mlsanity.com/wm/latent-dynamics-models-and-model-based-rl/)

## Builds on (most cited first, 30 of 35)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [CURL: Contrastive Unsupervised Representations for Reinforcement Learning](https://mlsanity.com/paper/2004.04136/) (2020)
- [Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels](https://mlsanity.com/paper/2004.13649/) (2020)
- [DensePose: Dense Human Pose Estimation In The Wild](https://mlsanity.com/paper/1802.00434/) (2018)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [Reinforcement Learning with Augmented Data](https://mlsanity.com/paper/2004.14990/) (2020)
- [Learning Invariant Representations for Reinforcement Learning without Reconstruction](https://mlsanity.com/paper/2006.10742/) (2020)
- [DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition](https://mlsanity.com/paper/1310.1531/) (2013)
- [INeRF: Inverting Neural Radiance Fields for Pose Estimation](https://mlsanity.com/paper/2012.05877/) (2020)
- [dm_control: Software and Tasks for Continuous Control](https://mlsanity.com/paper/2006.12983/) (2020)
- [Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning](https://mlsanity.com/paper/2107.09645/) (2021)
- [FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPs](https://mlsanity.com/paper/2006.10814/) (2020)
- [Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model](https://mlsanity.com/paper/1907.00953/) (2019)
- [DeepMDP: Learning Continuous Latent Space Models for Representation Learning](https://mlsanity.com/paper/1906.02736/) (2019)
- [What makes ImageNet good for transfer learning?](https://mlsanity.com/paper/1608.08614/) (2016)
- [Provably efficient RL with Rich Observations via Latent State Decoding](https://mlsanity.com/paper/1901.09018/) (2019)
- [kPAM: KeyPoint Affordances for Category-Level Robotic Manipulation](https://mlsanity.com/paper/1903.06684/) (2019)
- [Scalable methods for computing state similarity in deterministic Markov Decision Processes](https://mlsanity.com/paper/1911.09291/) (2019)
- [Sample Complexity of Reinforcement Learning using Linearly Combined Model Ensembles](https://mlsanity.com/paper/1910.10597/) (2019)
- [A Short Note on the Kinetics-700-2020 Human Action Dataset](https://mlsanity.com/paper/2010.10864/) (2020)
- [Learning to Simulate Dynamic Environments with GameGAN](https://mlsanity.com/paper/2005.12126/) (2020)
- [Mask-Guided Portrait Editing with Conditional GANs](https://mlsanity.com/paper/1905.10346/) (2019)

## Built on by (most cited first, 30 of 68)

- [Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learning](https://mlsanity.com/paper/2304.01203/) (2023)
- [Bridging State and History Representations: Understanding Self-Predictive RL](https://mlsanity.com/paper/2401.08898/) (2024)
- [Guaranteed Discovery of Control-Endogenous Latent States with Multi-Step Inverse Models](https://mlsanity.com/paper/2207.08229/) (2022)
- [Latent Action Learning Requires Supervision in the Presence of Distractors](https://mlsanity.com/paper/2502.00379/) (2025)
- [World Models and Predictive Coding for Cognitive and Developmental Robotics: Frontiers and Challenges](https://mlsanity.com/paper/2301.05832/) (2023)
- [CCIL: Continuity-based Data Augmentation for Corrective Imitation Learning](https://mlsanity.com/paper/2310.12972/) (2023)
- [Improved Representation of Asymmetrical Distances with Interval Quasimetric Embeddings](https://mlsanity.com/paper/2211.15120/) (2022)
- [Learning World Models with Identifiable Factorization](https://mlsanity.com/paper/2306.06561/) (2023)
- [HarmonyDream: Task Harmonization Inside World Models](https://mlsanity.com/paper/2310.00344/) (2023)
- [Learning Latent Dynamic Robust Representations for World Models](https://mlsanity.com/paper/2405.06263/) (2024)
- [Language Control Diffusion: Efficiently Scaling through Space, Time, and Tasks](https://mlsanity.com/paper/2210.15629/) (2022)
- [RePo: Resilient Model-Based Reinforcement Learning by Regularizing Posterior Predictability](https://mlsanity.com/paper/2309.00082/) (2023)
- [Agent-Controller Representations: Principled Offline RL with Rich Exogenous Information](https://mlsanity.com/paper/2211.00164/) (2022)
- [Structure in Deep Reinforcement Learning: A Survey and Open Problems](https://mlsanity.com/paper/2306.16021/) (2023)
- [Latent State Marginalization as a Low-cost Approach for Improving Exploration](https://mlsanity.com/paper/2210.00999/) (2022)
- [Multistep Inverse Is Not All You Need](https://mlsanity.com/paper/2403.11940/) (2024)
- [Learning Interactive World Model for Object-Centric Reinforcement Learning](https://mlsanity.com/paper/2511.02225/) (2025)
- [Ignorance is Bliss: Robust Control via Information Gating](https://mlsanity.com/paper/2303.06121/) (2023)
- [SeMAIL: Eliminating Distractors in Visual Imitation via Separated Models](https://mlsanity.com/paper/2306.10695/) (2023)
- [Simplifying Latent Dynamics with Softly State-Invariant World Models](https://mlsanity.com/paper/2401.17835/) (2024)
- [AD3: Implicit Action is the Key for World Models to Distinguish the Diverse Visual Distractors](https://mlsanity.com/paper/2403.09976/) (2024)
- [Towards Principled Representation Learning from Videos for Reinforcement Learning](https://mlsanity.com/paper/2403.13765/) (2024)
- [Four Principles for Physically Interpretable World Models](https://mlsanity.com/paper/2503.02143/) (2025)
- [Combining Reconstruction and Contrastive Methods for Multimodal Representations in RL](https://mlsanity.com/paper/2302.05342/) (2023)
- [Discrete Codebook World Models for Continuous Control](https://mlsanity.com/paper/2503.00653/) (2025)
- [Focus On What Matters: Separated Models For Visual-Based RL Generalization](https://mlsanity.com/paper/2410.10834/) (2024)
- [Policy-shaped prediction: avoiding distractions in model-based reinforcement learning](https://mlsanity.com/paper/2412.05766/) (2024)
- [Reimagination with Test-time Observation Interventions: Distractor-Robust World Model Predictions for Visual Model Predictive Control](https://mlsanity.com/paper/2506.16565/) (2025)
- [PhyLatent: Learning Dynamics-Relevant Representations for JEPA World Models](https://mlsanity.com/paper/2608.05720/) (2026)
- [Task Aware Dreamer for Task Generalization in Reinforcement Learning](https://mlsanity.com/paper/2303.05092/) (2023)

## Abstract (arXiv)

The ability to separate signal from noise, and reason with clean abstractions, is critical to intelligence. With this ability, humans can efficiently perform real world tasks without considering all possible nuisance factors.How can artificial agents do the same? What kind of information can agents safely discard as noises? In this work, we categorize information out in the wild into four types based on controllability and relation with reward, and formulate useful information as that which is both controllable and reward-relevant. This framework clarifies the kinds information removed by various prior work on representation learning in reinforcement learning (RL), and leads to our proposed approach of learning a Denoised MDP that explicitly factors out certain noise distractors. Extensive experiments on variants of DeepMind Control Suite and RoboDesk demonstrate superior performance of our denoised world model over using raw observations alone, and over prior works, across policy optimization control tasks as well as the non-control task of joint position regression.