mlsanityData to 7 Oct 2026

Bridging State and History Representations: Understanding Self-Predictive RL

arXiv 2401.08898 ↗First posted 17 Jan 2024cs.LG cs.AIcited by 52

Not on the map. Why?

Abstract (arXiv)

Representations are at the core of all deep reinforcement learning (RL) methods for both Markov decision processes (MDPs) and partially observable Markov decision processes (POMDPs). Many representation learning methods and theoretical frameworks have been developed to understand what constitutes an effective representation. However, the relationships between these methods and the shared properties among them remain unclear. In this paper, we show that many of these seemingly distinct methods and frameworks for state and history abstractions are, in fact, based on a common idea of self-predictive abstraction. Furthermore, we provide theoretical insights into the widely adopted objectives and optimization, such as the stop-gradient technique, in learning self-predictive representations. These findings together yield a minimalist algorithm to learn self-predictive representations for states and histories. We validate our theories by applying our algorithm to standard MDPs, MDPs with distractors, and POMDPs with sparse rewards. These findings culminate in a set of preliminary guidelines for RL practitioners.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). Builds on 81 of its 127 references that are in the map. First posted on arXiv 17 Jan 2024.

Builds on

81 of 127 references.

References · 81

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2016Layer Normalizationcited by 5,579
  5. 2015Continuous control with deep reinforcement learningcited by 3,279
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  7. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  8. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  9. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  10. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  11. 2023Mastering Diverse Domains through World Modelscited by 1,193
  12. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
38 more
  1. 2020Mastering Atari with Discrete World Modelscited by 990
  2. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  3. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  4. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  5. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  6. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  7. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  8. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  9. 2019On Variational Bounds of Mutual Informationcited by 550
  10. 2019Model-Based Reinforcement Learning for Ataricited by 495
  11. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 457
  12. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  13. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  14. 2022Temporal Difference Learning for Model Predictive Controlcited by 375
  15. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
  16. 2016Successor Features for Transfer in Reinforcement Learningcited by 331
  17. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 320
  18. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
  19. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
  20. 2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 266
  21. 2021Mastering Atari Games with Limited Datacited by 235
  22. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 228
  23. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 204
  24. 2016Value Iteration Networkscited by 158
  25. 2019Benchmarking Model-Based Reinforcement Learningcited by 158
  26. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 156
  27. 2018State Representation Learning for Control: An Overviewcited by 125
  28. 2017Value Prediction Networkcited by 117
  29. 2021Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPscited by 115
  30. 2020Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learningcited by 101
  31. 2021The Distracting Control Suite -- A Challenging Benchmark for Reinforcement Learning from Pixelscited by 97
  32. 2016The Predictron: End-To-End Learning and Planningcited by 92
  33. 2020The Value Equivalence Principle for Model-Based Reinforcement Learningcited by 88
  34. 2020Deep Reinforcement and InfoMax Learningcited by 79
  35. 2021DreamerPro: Reconstruction-Free Model-Based Reinforcement Learning with Prototypical Representationscited by 73
  36. 2023For SALE: State-Action Representation Learning for Deep Reinforcement Learningcited by 73
  37. 2016Deep Successor Reinforcement Learningcited by 72
  38. 2016Loss is its own Reward: Self-Supervision for Reinforcement Learningcited by 72

50 most cited of 81 All 81 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 21; 2026: 19.

52 papers.

Sort:

By area:

  1. 2024Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Makingcited by 53
  2. 2025Towards General-Purpose Model-Free Reinforcement Learningcited by 40
  3. 2025Latent Action Learning Requires Supervision in the Presence of Distractorscited by 40
  4. 2025Next-Latent Prediction Transformers Learn Compact World Modelscited by 29
  5. 2025A Survey of State Representation Learning for Deep Reinforcement Learningcited by 17
  6. 2024MAD-TD: Model-Augmented Data stabilizes High Update Ratio RLcited by 16
  7. 2024When does Self-Prediction help? Understanding Auxiliary Tasks in Reinforcement Learningcited by 13
  8. 2025Improving Transformer World Models for Data-Efficient RLcited by 13
  9. 2024Do Transformer World Models Give Better Policy Gradients?cited by 10
  10. 2024iQRL -- Implicitly Quantized Representations for Sample-efficient Reinforcement Learningcited by 9
  11. 2024Agent-state based policies in POMDPs: Beyond belief-state MDPscited by 9
  12. 2025Self-Predictive Representations for Combinatorial Generalization in Behavioral Cloningcited by 9
38 more
  1. 2024A Unifying Framework for Action-Conditional Self-Predictive Reinforcement Learningcited by 7
  2. 2025seq-JEPA: Autoregressive Predictive Learning of Invariant-Equivariant World Modelscited by 7
  3. 2025Intention-Conditioned Flow Occupancy Modelscited by 7
  4. 2025Habitizing Diffusion Planning for Efficient and Effective Decision Makingcited by 6
  5. 2025Hadamax Encoding: Elevating Performance in Model-Free Ataricited by 6
  6. 2025Relative Entropy Pathwise Policy Optimizationcited by 6
  7. 2024Uncertainty Representations in State-Space Layers for Deep Reinforcement Learning under Partial Observabilitycited by 5
  8. 2025A Theoretical Justification for Asymmetric Actor-Critic Algorithmscited by 5
  9. 2024An Empirical Study on the Power of Future Prediction in Partially Observable Environmentscited by 3
  10. 2024Representation Learning For Efficient Deep Multi-Agent Reinforcement Learningcited by 3
  11. 2025SR-Reward: Taking The Path More Traveledcited by 3
  12. 2025Measuring In-Context Computation Complexity via Hidden State Predictioncited by 3
  13. 2025AI in a vat: Fundamental limits of efficient world modelling for agent sandboxing and interpretabilitycited by 3
  14. 2024Dynamical-VAE-based Hindsight to Learn the Causal Dynamics of Factored-POMDPscited by 2
  15. 2025Calibrated Value-Aware Model Learning with Probabilistic Environment Modelscited by 2
  16. 2025Informed Asymmetric Actor-Critic: Leveraging Privileged Signals Beyond Full-State Accesscited by 2
  17. 2026Dreamer-CDP: Improving Reconstruction-free World Models Via Continuous Deterministic Representation Predictioncited by 2
  18. 2024Latent Linear Quadratic Regulator for Robotic Control Taskscited by 1
  19. 2025On the Limits of Tabular Hardness Metrics for Deep RL: A Study with the Pharos Benchmarkcited by 1
  20. 2025Rethinking the Role of Dynamic Sparse Training for Scalable Deep Reinforcement Learningcited by 1
  21. 2026Can We Really Learn One Representation to Optimize All Rewards?cited by 1
  22. 2026Neural ODE and SDE Models for Adaptation and Planning in Model-Based Reinforcement Learningcited by 1
  23. 2026Debiased Model-based Representations for Sample-efficient Continuous Controlcited by 1
  24. 2025Descriptive History Representations: Learning Representations by Answering Questionscited by 0
  25. 2025Convergence of regularized agent-state-based Q-learning in POMDPscited by 0
  26. 2026Squeezing More from the Stream : Learning Representation Online for Streaming Reinforcement Learningcited by 0
  27. 2026Temporal Consistency Improves Generalization in Contextual Offline Meta Reinforcement Learningcited by 0
  28. 2026Integrating Causal DAGs in Deep RL: Activating Minimal Markovian States with Multi-Order Exposurecited by 0
  29. 2026R2R2: Robust Representation for Intensive Experience Reuse via Redundancy Reduction in Self-Predictive Learningcited by 0
  30. 2026Representation Learning Enables Scalable Multitask Deep Reinforcement Learningcited by 0
  31. 2026NASDAQ: Normalized Observation Space Dynamics-Augmented Q-Learningcited by 0
  32. 2026Compositional Behavioral Semantics for State Abstraction in Reinforcement Learningcited by 0
  33. 2026What a World Model Represents Is Three Questionscited by 0
  34. 2026Reinformed Dreamer: An Asymmetric World Model Efficiently Trained through Latent Guidancecited by 0
  35. 2026V-Simba: Unleashing the Architectural Potential of RL in Visual Continuous Controlcited by 0
  36. 2026World models of environment, agent and joint agent-environment systemscited by 0
  37. 2026Minimal Recurrent Behavioral Memory for Imitation under Partial Observabilitycited by 0
  38. 2026Scalable In-Context Reinforcement Learning with Recurrent Algorithm Distillationcited by 0

50 most cited of 52 All 52 (JSON)