mlsanityData to 7 Oct 2026

DeepMDP: Learning Continuous Latent Space Models for Representation Learning

arXiv 1906.02736 ↗First posted 6 Jun 2019cs.LG stat.MLcited by 204

Not on the map. Why?

Cited by 25 in World models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Many reinforcement learning (RL) tasks provide the agent with high-dimensional observations that can be simplified into low-dimensional continuous states. To formalize this process, we introduce the concept of a DeepMDP, a parameterized latent space model that is trained via the minimization of two tractable losses: prediction of rewards and prediction of the distribution over next latent states. We show that the optimization of these objectives guarantees (1) the quality of the latent space as a representation of the state space and (2) the quality of the DeepMDP as a model of the environment. We connect these results to prior work in the bisimulation literature, and explore the use of a variety of metrics. Our theoretical findings are substantiated by the experimental result that a trained DeepMDP recovers the latent structure underlying high-dimensional observations on a synthetic environment. Finally, we show that learning a DeepMDP as an auxiliary task in the Atari 2600 domain leads to large performance improvements over model-free RL.

Key facts. Cited by 204 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 2; 2021: 40; 2022: 45; 2023: 37; 2024: 27; 2025: 20; 2026: 33.

204 papers.

Sort:

By area:

  1. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  2. 2022Multi-Game Decision Transformerscited by 212
  3. 2022Masked World Models for Visual Controlcited by 169
  4. 2023Language-Driven Representation Learning for Roboticscited by 163
  5. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  6. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 124
  7. 2020Primal Wasserstein Imitation Learningcited by 120
  8. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 107
  9. 2021Pretraining Representations for Data-Efficient Reinforcement Learningcited by 105
  10. 2020Offline Reinforcement Learning from Images with Latent Space Modelscited by 94
  11. 2021Representation Matters: Offline Pretraining for Sequential Decision Makingcited by 90
  12. 2021Model-free Representation Learning and Exploration in Low-rank MDPscited by 82
38 more
  1. 2023Multi-View Masked World Models for Visual Robotic Manipulationcited by 78
  2. 2021DreamerPro: Reconstruction-Free Model-Based Reinforcement Learning with Prototypical Representationscited by 73
  3. 2023For SALE: State-Action Representation Learning for Deep Reinforcement Learningcited by 73
  4. 2022Denoised MDPs: Learning World Models Better Than the World Itselfcited by 66
  5. 2021Learning Task Informed Abstractionscited by 58
  6. 2022Pre-Trained Image Encoder for Generalizable Visual Reinforcement Learningcited by 58
  7. 2021On The Effect of Auxiliary Tasks on Representation Dynamicscited by 56
  8. 2023TACO: Temporal Latent Action-Driven Contrastive Loss for Visual Reinforcement Learningcited by 56
  9. 2024Bridging State and History Representations: Understanding Self-Predictive RLcited by 52
  10. 2022Imitation Learning by Estimating Expertise of Demonstratorscited by 50
  11. 2022Making Linear MDPs Practical via Contrastive Representation Learningcited by 49
  12. 2023Pre-training Contextualized World Models with In-the-wild Videos for Reinforcement Learningcited by 45
  13. 2021MICo: Improved representations via sampling-based state similarity for Markov decision processescited by 43
  14. 2022Visuo-Tactile Transformers for Manipulationcited by 43
  15. 2021Learning Markov State Abstractions for Deep Reinforcement Learningcited by 42
  16. 2021Visual Adversarial Imitation Learning using Variational Modelscited by 42
  17. 2022Contextualize Me -- The Case for Context in Reinforcement Learningcited by 41
  18. 2023Proto-Value Networks: Scaling Representation Learning with Auxiliary Taskscited by 41
  19. 2025Towards General-Purpose Model-Free Reinforcement Learningcited by 40
  20. 2021TRAIL: Near-Optimal Imitation Learning with Suboptimal Datacited by 39
  21. 2022Bisimulation Makes Analogies in Goal-Conditioned Reinforcement Learningcited by 38
  22. 2022Learning to Accelerate Partial Differential Equations via Latent Global Evolutioncited by 38
  23. 2021Return-Based Contrastive Representation Learning for Reinforcement Learningcited by 36
  24. 2022Spectral Decomposition Representation for Reinforcement Learningcited by 35
  25. 2021Provable Representation Learning for Imitation with Contrastive Fourier Featurescited by 34
  26. 2021Proper Value Equivalencecited by 33
  27. 2024The Essential Role of Causality in Foundation World Models for Embodied AIcited by 32
  28. 2021Action-Sufficient State Representation Learning for Control with Structural Constraintscited by 31
  29. 2023Inverse Dynamics Pretraining Learns Good Representations for Multitask Imitationcited by 30
  30. 2021Robust Predictable Controlcited by 29
  31. 2025Next-Latent Prediction Transformers Learn Compact World Modelscited by 29
  32. 2021Offline Reinforcement Learning with Pseudometric Learningcited by 28
  33. 2021Which Mutual-Information Representation Learning Objectives are Sufficient for Control?cited by 28
  34. 2021Provable RL with Exogenous Distractors via Multistep Inverse Dynamicscited by 28
  35. 2021Koopman Q-learning: Offline Reinforcement Learning via Symmetries of Dynamicscited by 27
  36. 2022INFOrmation Prioritization through EmPOWERment in Visual Model-Based RLcited by 27
  37. 2021Metrics and continuity in reinforcement learningcited by 26
  38. 2022Sample-Efficient Reinforcement Learning in the Presence of Exogenous Informationcited by 26

50 most cited of 204 All 204 (JSON)