mlsanityData to 7 Oct 2026

Denoised MDPs: Learning World Models Better Than the World Itself

arXiv 2206.15477 ↗First posted 30 Jun 2022cs.LGcited by 66

On the map: World models → Latent dynamics models and model-based RL

Abstract (arXiv)

The ability to separate signal from noise, and reason with clean abstractions, is critical to intelligence. With this ability, humans can efficiently perform real world tasks without considering all possible nuisance factors.How can artificial agents do the same? What kind of information can agents safely discard as noises? In this work, we categorize information out in the wild into four types based on controllability and relation with reward, and formulate useful information as that which is both controllable and reward-relevant. This framework clarifies the kinds information removed by various prior work on representation learning in reinforcement learning (RL), and leads to our proposed approach of learning a Denoised MDP that explicitly factors out certain noise distractors. Extensive experiments on variants of DeepMind Control Suite and RoboDesk demonstrate superior performance of our denoised world model over using raw observations alone, and over prior works, across policy optimization control tasks as well as the non-control task of joint position regression.

Key facts. Cited by 66 papers in the map (data through 7 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 35 of its 55 references that are in the map. First posted on arXiv 30 Jun 2022.

Builds on

35 of 55 references.

Closest ancestors · 2

Same direction, newest first

  1. 2019Dream to Control: Learning Behaviors by Latent Imagination3 Dec
  2. 2018World Models27 Mar

Other references · 33

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  2. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  3. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  4. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  5. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  6. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  7. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  8. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  9. 2018DensePose: Dense Human Pose Estimation In The Wildcited by 573
  10. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  11. 2019On Variational Bounds of Mutual Informationcited by 550
  12. 2020Reinforcement Learning with Augmented Datacited by 452
21 more
  1. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  2. 2013DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognitioncited by 422
  3. 2020INeRF: Inverting Neural Radiance Fields for Pose Estimationcited by 381
  4. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  5. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  6. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 232
  7. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 228
  8. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 204
  9. 2016What makes ImageNet good for transfer learning?cited by 192
  10. 2019Provably efficient RL with Rich Observations via Latent State Decodingcited by 169
  11. 2019kPAM: KeyPoint Affordances for Category-Level Robotic Manipulationcited by 152
  12. 2019Scalable methods for computing state similarity in deterministic Markov Decision Processescited by 133
  13. 2019Sample Complexity of Reinforcement Learning using Linearly Combined Model Ensemblescited by 100
  14. 2020A Short Note on the Kinetics-700-2020 Human Action Datasetcited by 89
  15. 2020Learning to Simulate Dynamic Environments with GameGANcited by 86
  16. 2019Mask-Guided Portrait Editing with Conditional GANscited by 61
  17. 2021Learning Task Informed Abstractionscited by 58
  18. 2020Predictive Information Accelerates Learning in RLcited by 56
  19. 2019A Geometric Perspective on Optimal Representations for Reinforcement Learningcited by 54
  20. 2021Robust Predictable Controlcited by 29
  21. 2021Provable RL with Exogenous Distractors via Multistep Inverse Dynamicscited by 28

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 7; 2023: 13; 2024: 18; 2025: 12; 2026: 16.

66 papers.

Sort:

By area:

  1. 2023Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learningcited by 116
  2. 2024Bridging State and History Representations: Understanding Self-Predictive RLcited by 52
  3. 2022Guaranteed Discovery of Control-Endogenous Latent States with Multi-Step Inverse Modelscited by 42
  4. 2025Latent Action Learning Requires Supervision in the Presence of Distractorscited by 40
  5. 2023World Models and Predictive Coding for Cognitive and Developmental Robotics: Frontiers and Challengescited by 35
  6. 2023CCIL: Continuity-based Data Augmentation for Corrective Imitation Learningcited by 26
  7. 2023Learning World Models with Identifiable Factorizationcited by 23
  8. 2022Improved Representation of Asymmetrical Distances with Interval Quasimetric Embeddingscited by 22
  9. 2023HarmonyDream: Task Harmonization Inside World Modelscited by 22
  10. 2022Language Control Diffusion: Efficiently Scaling through Space, Time, and Taskscited by 20
  11. 2024Learning Latent Dynamic Robust Representations for World Modelscited by 20
  12. 2023RePo: Resilient Model-Based Reinforcement Learning by Regularizing Posterior Predictabilitycited by 17
38 more
  1. 2023Structure in Deep Reinforcement Learning: A Survey and Open Problemscited by 16
  2. 2022Agent-Controller Representations: Principled Offline RL with Rich Exogenous Informationcited by 15
  3. 2022Latent State Marginalization as a Low-cost Approach for Improving Explorationcited by 13
  4. 2025Learning Interactive World Model for Object-Centric Reinforcement Learningcited by 11
  5. 2023Ignorance is Bliss: Robust Control via Information Gatingcited by 10
  6. 2023SeMAIL: Eliminating Distractors in Visual Imitation via Separated Modelscited by 10
  7. 2024Simplifying Latent Dynamics with Softly State-Invariant World Modelscited by 10
  8. 2024AD3: Implicit Action is the Key for World Models to Distinguish the Diverse Visual Distractorscited by 10
  9. 2024Multistep Inverse Is Not All You Needcited by 10
  10. 2024Towards Principled Representation Learning from Videos for Reinforcement Learningcited by 10
  11. 2025Four Principles for Physically Interpretable World Modelscited by 10
  12. 2023Combining Reconstruction and Contrastive Methods for Multimodal Representations in RLcited by 9
  13. 2025Discrete Codebook World Models for Continuous Controlcited by 9
  14. 2024Policy-shaped prediction: avoiding distractions in model-based reinforcement learningcited by 6
  15. 2025Reimagination with Test-time Observation Interventions: Distractor-Robust World Model Predictions for Visual Model Predictive Controlcited by 6
  16. 2023Task Aware Dreamer for Task Generalization in Reinforcement Learningcited by 5
  17. 2024Focus On What Matters: Separated Models For Visual-Based RL Generalizationcited by 5
  18. 2023Model-Based Reinforcement Learning with Isolated Imaginationscited by 4
  19. 2024Make the Pertinent Salient: Task-Relevant Reconstruction for Visual Control with Distractionscited by 4
  20. 2025Towards Empowerment Gain through Causal Structure Learning in Model-Based RLcited by 4
  21. 2026PhyLatent: Learning Dynamics-Relevant Representations for JEPA World Modelscited by 4
  22. 2022Cost-Driven Representation Learning for Linear Quadratic Gaussian Control: Part Icited by 3
  23. 2024BECAUSE: Bilinear Causal Representation for Generalizable Offline Model-based Reinforcement Learningcited by 3
  24. 2024Learning a Fast Mixing Exogenous Block MDP using a Single Trajectorycited by 3
  25. 2025Cloning Deterministic Worlds: The Critical Role of Latent Geometry in Long-Horizon World Modelscited by 3
  26. 2022Disentangled (Un)Controllable Featurescited by 2
  27. 2024Generalizing Multi-Step Inverse Models for Representation Learning to Finite-Memory POMDPscited by 2
  28. 2025DyMoDreamer: World Modeling with Dynamic Modulationcited by 2
  29. 2025From Pixels to Factors: Learning Independently Controllable State Variables for Reinforcement Learningcited by 2
  30. 2026Discovering What You Can Control: Interventional Boundary Discovery for Reinforcement Learningcited by 2
  31. 2026Learning Task-Sufficient World Models by Synergizing Agentic Exploration and Structured Modelingcited by 2
  32. 2026DWM: Separating World Effects from Actions in Latent World Modelscited by 2
  33. 2026Diagnosing JEPA World Models with Action-Conditioned Predictive Consistencycited by 2
  34. 2023VIBR: Learning View-Invariant Value Functions for Robust Visual Controlcited by 1
  35. 2024Reward-Relevance-Filtered Linear Offline Reinforcement Learningcited by 1
  36. 2024Learning Off-policy with Model-based Intrinsic Motivation For Active Online Explorationcited by 1
  37. 2024SeMOPO: Learning High-quality Model and Policy from Low-quality Offline Visual Datasetscited by 1
  38. 2024Abstract Reward Processes: Leveraging State Abstraction for Consistent Off-Policy Evaluationcited by 1

50 most cited of 66 All 66 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Reinforcement Learning from Denoising Feedback
  2. 2021A Closer Look at Advantage-Filtered Behavioral Cloning in High-Noise Datasets
  3. 2025D3P: Dynamic Denoising Diffusion Policy via Reinforcement Learning
  4. 2026DenoiseRL: Bootstrapping Reasoning Models to Recover from Noisy Prefixes
  5. 2021Information is Power: Intrinsic Control via Information Capture
  6. 2026Noisy-Space Policy Gradient for Diffusion Policies in Offline Reinforcement Learning
  7. 2025Characterizing higher-order representations through generative diffusion models explains human decoded neurofeedback performance
  8. 2025MDPO: Overcoming the Training-Inference Divide of Masked Diffusion Language Models