mlsanityData to 7 Oct 2026

Learning Markov State Abstractions for Deep Reinforcement Learning

arXiv 2106.04379 ↗First posted 8 Jun 2021cs.LG cs.AI stat.MLcited by 42

Not on the map. Why?

Abstract (arXiv)

A fundamental assumption of reinforcement learning in Markov decision processes (MDPs) is that the relevant decision process is, in fact, Markov. However, when MDPs have rich observations, agents typically learn by way of an abstract state representation, and such representations are not guaranteed to preserve the Markov property. We introduce a novel set of conditions and prove that they are sufficient for learning a Markov abstract state representation. We then describe a practical training procedure that combines inverse model estimation and temporal contrastive learning to learn an abstraction that approximately satisfies these conditions. Our novel training objective is compatible with both online and offline training: it does not require a reward signal, but agents can capitalize on reward information when available. We empirically evaluate our approach on a visual gridworld domain and a set of continuous control benchmarks. Our approach learns representations that capture the underlying structure of the domain and lead to improved sample efficiency over state-of-the-art deep reinforcement learning with visual features -- often matching or exceeding the performance achieved with hand-designed compact state information.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Builds on 35 of its 64 references that are in the map. First posted on arXiv 8 Jun 2021.

Builds on

35 of 64 references.

References · 35

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  4. 2018World Modelscited by 1,444
  5. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  6. 2018Exploration by Random Network Distillationcited by 966
  7. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  8. 2019Model-Based Reinforcement Learning for Ataricited by 495
  9. 2020Reinforcement Learning with Augmented Datacited by 452
  10. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  11. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  12. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
23 more
  1. 2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 266
  2. 2020Decoupling Representation Learning from Reinforcement Learningcited by 263
  3. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 228
  4. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 204
  5. 2021Reinforcement Learning with Prototypical Representationscited by 179
  6. 2019Provably efficient RL with Rich Observations via Latent State Decodingcited by 169
  7. 2016Learning to Poke by Poking: Experiential Learning of Intuitive Physicscited by 152
  8. 2015Deep Spatial Autoencoders for Visuomotor Learningcited by 149
  9. 2017DARLA: Improving Zero-Shot Transfer in Reinforcement Learningcited by 147
  10. 2019Unsupervised State Representation Learning in Ataricited by 144
  11. 2019Scalable methods for computing state similarity in deterministic Markov Decision Processescited by 133
  12. 2019When to use parametric models in reinforcement learning?cited by 118
  13. 2019Kinematic State Abstraction and Provably Efficient Rich-Observation Reinforcement Learningcited by 114
  14. 2018Lipschitz Continuity in Model-based Reinforcement Learningcited by 90
  15. 2016Loss is its own Reward: Self-Supervision for Reinforcement Learningcited by 72
  16. 2017Near Optimal Behavior via Approximate State Abstractioncited by 63
  17. 2020Plannable Approximations to MDP Homomorphisms: Equivariance under Actionscited by 61
  18. 2018Learning Plannable Representations with Causal InfoGANcited by 56
  19. 2016Transfer from Simulation to Real World through Learning Deep Inverse Dynamics Modelcited by 53
  20. 2018Contingency-Aware Exploration in Reinforcement Learningcited by 26
  21. 2018Decoupling Dynamics and Reward for Transfer Learningcited by 23
  22. 2018Efficient Model-Based Deep Reinforcement Learning with Variational State Tabulationcited by 20
  23. 2020Learning Discrete State Abstractions With Deep Variational Inferencecited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 4; 2023: 8; 2024: 11; 2025: 12; 2026: 6.

42 papers.

Sort:

By area:

  1. 2024Bridging State and History Representations: Understanding Self-Predictive RLcited by 52
  2. 2023Flipping Coins to Estimate Pseudocounts for Exploration in Reinforcement Learningcited by 28
  3. 2023Conditional Mutual Information for Disentangled Representations in Reinforcement Learningcited by 27
  4. 2022Unsupervised Representation Learning in Deep Reinforcement Learning: A Reviewcited by 18
  5. 2022Pretraining in Deep Reinforcement Learning: A Surveycited by 18
  6. 2025A Survey of State Representation Learning for Deep Reinforcement Learningcited by 17
  7. 2023Structure in Deep Reinforcement Learning: A Survey and Open Problemscited by 16
  8. 2024Inference via Interpolation: Contrastive Representations Provably Enable Planning and Inferencecited by 15
  9. 2021SimSR: Simple Distance-based State Representation for Deep Reinforcement Learningcited by 14
  10. 2023Hierarchical State Abstraction Based on Structural Information Principlescited by 14
  11. 2025One Subgoal at a Time: Zero-Shot Generalization to Arbitrary Linear Temporal Logic Requirements in Multi-Task Reinforcement Learningcited by 8
  12. 2022Meta-Learning Parameterized Skillscited by 7
30 more
  1. 2023Sequential Knockoffs for Variable Selection in Reinforcement Learningcited by 7
  2. 2024Mitigating Partial Observability in Sequential Decision Processes via the Lambda Discrepancycited by 7
  3. 2025Studying the Interplay Between the Actor and Critic Representations in Reinforcement Learningcited by 7
  4. 2025Temporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learningcited by 5
  5. 2024Learning Abstract World Model for Value-preserving Planning with Optionscited by 4
  6. 2024BAMDP Shaping: a Unified Framework for Intrinsic Motivation and Reward Shapingcited by 4
  7. 2026Improving Policy Exploitation in Online Reinforcement Learning with Instant Retrospect Actioncited by 4
  8. 2022An Analysis of Model-Based Reinforcement Learning From Abstracted Observationscited by 2
  9. 2023Representations and Exploration for Deep Reinforcement Learning using Singular Value Decompositioncited by 2
  10. 2024Off-policy Evaluation with Deeply-abstracted Statescited by 2
  11. 2025Beyond Non-Expert Demonstrations: Outcome-Driven Action Constraint for Offline Reinforcement Learningcited by 2
  12. 2025From Pixels to Factors: Learning Independently Controllable State Variables for Reinforcement Learningcited by 2
  13. 2023Contextual Conservative Q-Learning for Offline Reinforcement Learningcited by 1
  14. 2024On shallow planning under partial observabilitycited by 1
  15. 2024Abstract Reward Processes: Leveraging State Abstraction for Consistent Off-Policy Evaluationcited by 1
  16. 2024The Synergy of LLMs & RL Unlocks Offline Learning of Generalizable Language-Conditioned Policies with Low-fidelity Datacited by 1
  17. 2025Inversely Learning Transferable Rewards via Abstracted Statescited by 1
  18. 2025Multi-Agent Conditional Diffusion Model with Mean Field Communication as Wireless Resource Allocation Plannercited by 1
  19. 2026Toward Active Object Detection for UAVs in the Wild: A Large-Scale Dataset, Benchmark and Methodcited by 1
  20. 2023Goal-oriented inference of environment from redundant observationscited by 0
  21. 2024CrowdTransfer: Enabling Crowd Knowledge Transfer in AIoT Communitycited by 0
  22. 2024Localized Observation Abstraction Using Piecewise Linear Spatial Decay for Reinforcement Learning in Combat Simulationscited by 0
  23. 2025Task-Driven Discrete Representation Learningcited by 0
  24. 2025Pretraining in Actor-Critic Reinforcement Learning for Locomotioncited by 0
  25. 2025Realizable Abstractions: Near-Optimal Hierarchical Reinforcement Learningcited by 0
  26. 2025Horizon Reduction as Information Loss in Offline Reinforcement Learningcited by 0
  27. 2026Mask-based Predictive Representations for Reinforcement Learningcited by 0
  28. 2026RTPO: Reverse-Turn Policy Optimization for Stabilizing Agentic RL Trainingcited by 0
  29. 2026AD-WM: Action-Discriminative World Models for Counterfactual Model Predictive Controlcited by 0
  30. 2026Going Beyond State-Reaching: Learning Abstractions for Intrinsically Motivated Option Discoverycited by 0