mlsanityData to 14 Oct 2026

Studying the Interplay Between the Actor and Critic Representations in Reinforcement Learning

arXiv 2503.06343 ↗First posted 8 Mar 2025cs.LG cs.AIcited by 8

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Extracting relevant information from a stream of high-dimensional observations is a central challenge for deep reinforcement learning agents. Actor-critic algorithms add further complexity to this challenge, as it is often unclear whether the same information will be relevant to both the actor and the critic. To this end, we here explore the principles that underlie effective representations for the actor and for the critic in on-policy algorithms. We focus our study on understanding whether the actor and critic will benefit from separate, rather than shared, representations. Our primary finding is that when separated, the representations for the actor and critic systematically specialise in extracting different types of information from the environment -- the actor's representation tends to focus on action-relevant information, while the critic's representation specialises in encoding value and dynamics information. We conduct a rigourous empirical study to understand how different representation learning approaches affect the actor and critic's specialisations and their downstream performance, in terms of sample efficiency and generation capabilities. Finally, we discover that a separated critic plays an important role in exploration and data collection during training. Our code, trained models and data are accessible at https://github.com/francelico/deac-rep.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 40 of its 50 references that are in the map. First posted on arXiv 8 Mar 2025.

Builds on

40 of 50 references.

References · 40

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  4. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  5. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  6. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 483
  7. 2020Reinforcement Learning with Augmented Datacited by 468
  8. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 436
  9. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 404
  10. 2020Planning to Explore via Self-Supervised World Modelscited by 368
  11. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 318
  12. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 274
28 more
  1. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 207
  2. 2020Prioritized Level Replaycited by 198
  3. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 170
  4. 2017DARLA: Improving Zero-Shot Transfer in Reinforcement Learningcited by 149
  5. 2019Unsupervised State Representation Learning in Ataricited by 144
  6. 2019Scalable methods for computing state similarity in deterministic Markov Decision Processescited by 133
  7. 2020Phasic Policy Gradientcited by 123
  8. 2021Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learningcited by 123
  9. 2021The Distracting Control Suite -- A Challenging Benchmark for Reinforcement Learning from Pixelscited by 100
  10. 2021Decoupling Value and Policy for Generalization in Reinforcement Learningcited by 82
  11. 2019Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learningcited by 60
  12. 2020Predictive Information Accelerates Learning in RLcited by 56
  13. 2021Learning Markov State Abstractions for Deep Reinforcement Learningcited by 44
  14. 2021MICo: Improved representations via sampling-based state similarity for Markov decision processescited by 43
  15. 2018Neural Predictive Belief Representationscited by 40
  16. 2023Conditional Mutual Information for Disentangled Representations in Reinforcement Learningcited by 27
  17. 2022Investigating the Properties of Neural Network Representations in Reinforcement Learningcited by 24
  18. 2021Domain Adversarial Reinforcement Learningcited by 19
  19. 2022Temporal Disentanglement of Representations for Improved Generalisation in Reinforcement Learningcited by 19
  20. 2020Instance based Generalization in Reinforcement Learningcited by 17
  21. 2024DRED: Zero-Shot Transfer in Reinforcement Learning via Data-Regularised Environment Designcited by 11
  22. 2020Are Gradient-based Saliency Maps Useful in Deep Reinforcement Learning?cited by 9
  23. 2022Rethinking Value Function Learning for Generalization in Reinforcement Learningcited by 8
  24. 2022Multi-Horizon Representations with Hierarchical Forward Models for Reinforcement Learningcited by 7
  25. 2019Supervise Thyself: Examining Self-Supervised Representations in Interactive Environmentscited by 4
  26. 2023Planning to Go Out-of-Distribution in Offline-to-Online Reinforcement Learningcited by 4
  27. 2023A Kernel Perspective on Behavioural Metrics for Markov Decision Processescited by 3
  28. 2025PixelBrax: Learning Continuous Control from Pixels End-to-End on the GPUcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 2.

8 papers.

  1. 2025Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agentscited by 12
  2. 2025Enhancing Tactile-based Reinforcement Learning for Robotic Controlcited by 5
  3. 2025Self-Supervised Multisensory Pretraining for Contact-Rich Robot Reinforcement Learningcited by 3
  4. 2026MO-Playground: Massively Parallelized Multi-Objective Reinforcement Learning for Roboticscited by 2
  5. 2025Stackelberg Coupling of Online Representation Learning and Reinforcement Learningcited by 1
  6. 2026Action-Sufficient Goal Representationscited by 1
  7. 2025Remembering the Markov Property in Cooperative MARLcited by 0
  8. 2025A New Perspective on Transformers in Online Reinforcement Learning for Continuous Controlcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Optimistic critics can empower small actors
  2. 2026Simple Actors and Deep Critics for Scalable Reinforcement Learning
  3. 2025Functional Critics Are Essential for Actor-Critic: From Off-Policy Stability to Efficient Exploration
  4. 2025Informed Asymmetric Actor-Critic: Leveraging Privileged Signals Beyond Full-State Access
  5. 2025A Survey of State Representation Learning for Deep Reinforcement Learning
  6. 2024Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning
  7. 2024A Role of Environmental Complexity on Representation Learning in Deep Reinforcement Learning Agents
  8. 2026Learning Suffers More Than the Policy Class Under Partial Observability: A Closed-Form Analysis