mlsanityData to 14 Oct 2026

A Survey of State Representation Learning for Deep Reinforcement Learning

arXiv 2506.17518 ↗First posted 20 Jun 2025cs.LG cs.AI stat.MLcited by 17

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Representation learning methods are an important tool for addressing the challenges posed by complex observations spaces in sequential decision making problems. Recently, many methods have used a wide variety of types of approaches for learning meaningful state representations in reinforcement learning, allowing better sample efficiency, generalization, and performance. This survey aims to provide a broad categorization of these methods within a model-free online setting, exploring how they tackle the learning of state representations differently. We categorize the methods into six main classes, detailing their mechanisms, benefits, and limitations. Through this taxonomy, our aim is to enhance the understanding of this field and provide a guide for new researchers. We also discuss techniques for assessing the quality of representations, and detail relevant future directions.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 120 of its 150 references that are in the map. First posted on arXiv 20 Jun 2025.

Builds on

120 of 150 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Towards General-Purpose Model-Free Reinforcement Learning27 Jan
  2. 2024State Chrono Representation for Enhancing Generalization in Reinforcement Learning9 Nov
  3. 2024When does Self-Prediction help? Understanding Auxiliary Tasks in Reinforcement Learning25 Jun
  4. 2024A Unifying Framework for Action-Conditional Self-Predictive Reinforcement Learning4 Jun
  5. 2024Learning Action-based Representations Using Invariance25 Mar
  6. 2024Towards Principled Representation Learning from Videos for Reinforcement Learning20 Mar
  7. 2024Foundation Policies with Hilbert Representations23 Feb
  8. 2024Zero-Shot Reinforcement Learning via Function Encoders30 Jan
  9. 2024Bridging State and History Representations: Understanding Self-Predictive RL17 Jan

Other references · 111

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  2. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  3. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  5. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  6. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  7. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  8. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,273
  9. 2017Hindsight Experience Replaycited by 1,136
  10. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  11. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  12. 2016Deep Variational Information Bottleneckcited by 930
38 more
  1. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  2. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  3. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  4. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  5. 2020Reinforcement Learning with Augmented Datacited by 468
  6. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 459
  7. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 436
  8. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
  9. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 318
  10. 2016Contextual Decision Processes with Low Bellman Rank are PAC-Learnablecited by 292
  11. 2020Decoupling Representation Learning from Reinforcement Learningcited by 269
  12. 2022Masked Visual Pre-training for Motor Controlcited by 248
  13. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 218
  14. 2021Multi-Task Reinforcement Learning with Context-based Representationscited by 174
  15. 2019Provably efficient RL with Rich Observations via Latent State Decodingcited by 169
  16. 2022RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rankcited by 153
  17. 2017DARLA: Improving Zero-Shot Transfer in Reinforcement Learningcited by 149
  18. 2019Unsupervised State Representation Learning in Ataricited by 144
  19. 2017Visualizing and Understanding Atari Agentscited by 141
  20. 2018Multi-task Deep Reinforcement Learning with PopArtcited by 141
  21. 2019Scalable methods for computing state similarity in deterministic Markov Decision Processescited by 133
  22. 2018State Representation Learning for Control: An Overviewcited by 127
  23. 2021Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learningcited by 123
  24. 2023Learning to Act without Actionscited by 121
  25. 2021Learning One Representation to Optimize All Rewardscited by 115
  26. 2021Pretraining Representations for Data-Efficient Reinforcement Learningcited by 105
  27. 2021Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentationcited by 105
  28. 2020Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learningcited by 102
  29. 2019Generalization in Reinforcement Learning with Selective Noise Injection and Information Bottleneckcited by 101
  30. 2018Adapting Auxiliary Losses Using Gradient Similaritycited by 100
  31. 2021Representation Matters: Offline Pretraining for Sequential Decision Makingcited by 91
  32. 2018The Laplacian in RL: Learning Representations with Efficient Approximationscited by 86
  33. 2022Does Zero-Shot Reinforcement Learning Exist?cited by 83
  34. 2019Towards Interpretable Reinforcement Learning Using Attention Augmented Agentscited by 82
  35. 2020Deep Reinforcement and InfoMax Learningcited by 79
  36. 2019Learning Action Representations for Reinforcement Learningcited by 75
  37. 2023For SALE: State-Action Representation Learning for Deep Reinforcement Learningcited by 75
  38. 2024Learning and Leveraging World Models in Visual Representation Learningcited by 72

50 most cited of 111 All 111 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 13.

17 papers.

Sort:

By area:

  1. 2025Dual Goal Representationscited by 16
  2. 2026Stable Deep Reinforcement Learning via Isotropic Gaussian Representationscited by 5
  3. 2025Skill Learning via Policy Diversity Yields Identifiable Representations for Reinforcement Learningcited by 4
  4. 2025Deep SPI: Safe Policy Improvement via World Modelscited by 3
  5. 2025PvP: Data-Efficient Humanoid Robot Learning with Proprioceptive-Privileged Contrastive Representationscited by 2
  6. 2026Squint: Fast Visual Reinforcement Learning for Sim-to-Real Roboticscited by 2
  7. 2026Abstraction for Offline Goal-Conditioned Reinforcement Learningcited by 2
  8. 2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learningcited by 2
  9. 2026MSPR: Multi-scale Predictive Representations for Goal-conditioned Reinforcement Learningcited by 1
  10. 2026State Design Matters: How Representations Shape Dynamic Reasoning in Large Language Modelscited by 0
  11. 2026Pixel-level Scene Understanding in One Token: Visual States Need What-is-Where Compositioncited by 0
  12. 2026Belief-State RWKV for Reinforcement Learning under Partial Observabilitycited by 0
5 more
  1. 2026Learning in Low-Dimensional Subspaces: Orthogonal Bottlenecks for Reinforcement Learningcited by 0
  2. 2026Representation Learning Enables Scalable Multitask Deep Reinforcement Learningcited by 0
  3. 2026A Unified Causal-Origin Taxonomy of Distributional Shifts in Reinforcement Learningcited by 0
  4. 2026Compositional Behavioral Semantics for State Abstraction in Reinforcement Learningcited by 0
  5. 2026V-Simba: Unleashing the Architectural Potential of RL in Visual Continuous Controlcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Harnessing Discrete Representations For Continual Reinforcement Learning
  2. 2024A Comprehensive Survey of Reinforcement Learning: From Algorithms to Practical Challenges
  3. 2025Reinforcement Learning for Large Model: A Survey
  4. 2024Efficient Exploration in Deep Reinforcement Learning: A Novel Bayesian Actor-Critic Algorithm
  5. 2022Behavior Prior Representation learning for Offline Reinforcement Learning
  6. 2025Reward Models in Deep Reinforcement Learning: A Survey
  7. 2021Provably Efficient Representation Selection in Low-rank Markov Decision Processes: From Online to Offline RL
  8. 2024The Definitive Guide to Policy Gradients in Deep Reinforcement Learning: Theory, Algorithms and Implementations