mlsanityData to 14 Oct 2026

A Theoretical Justification for Asymmetric Actor-Critic Algorithms

arXiv 2501.19116 ↗First posted 31 Jan 2025cs.LG stat.MLcited by 6

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

In reinforcement learning for partially observable environments, many successful algorithms have been developed within the asymmetric learning paradigm. This paradigm leverages additional state information available at training time for faster learning. Although the proposed learning objectives are usually theoretically sound, these methods still lack a precise theoretical justification for their potential benefits. We propose such a justification for asymmetric actor-critic algorithms with linear function approximators by adapting a finite-time convergence analysis to this setting. The resulting finite-time bound reveals that the asymmetric critic eliminates error terms arising from aliasing in the agent state.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 30 of its 45 references that are in the map. First posted on arXiv 31 Jan 2025.

Builds on

30 of 45 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Provable Partially Observable Reinforcement Learning with Privileged Information1 Dec
  2. 2024Agent-state based policies in POMDPs: Beyond belief-state MDPs24 Sep
  3. 2024Recurrent Natural Policy Gradient for POMDPs28 May

Other references · 27

  1. 2015Trust Region Policy Optimizationcited by 2,580
  2. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  3. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  4. 2019Solving Rubik's Cube with a Robot Handcited by 734
  5. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 465
  6. 2017Asymmetric Actor Critic for Image-Based Robot Learningcited by 238
  7. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 231
  8. 2018Deep Variational Reinforcement Learning for POMDPscited by 125
  9. 2020Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learningcited by 102
  10. 2015Memory-based control with recurrent neural networkscited by 78
  11. 2020Approximate information state for approximate planning and reinforcement learning in partially observed systemscited by 70
  12. 2019Shaping Belief States with Generative Environment Models for RLcited by 66
15 more
  1. 2024Bridging State and History Representations: Understanding Self-Predictive RLcited by 54
  2. 2018On Improving Deep Reinforcement Learning for POMDPscited by 50
  3. 2018Learning and Querying Fast Generative Models for Reinforcement Learningcited by 48
  4. 2019Variational Recurrent Models for Solving Partially Observable Control Taskscited by 47
  5. 2021Simple Agent, Complex Environment: Efficient Reinforcement Learning with Agent Statescited by 43
  6. 2018Neural Predictive Belief Representationscited by 40
  7. 2017Data-driven Planning via Imitation Learningcited by 36
  8. 2021Unbiased Asymmetric Reinforcement Learning under Partial Observabilitycited by 25
  9. 2020Robust Asymmetric Learning in POMDPscited by 23
  10. 2024A Super-human Vision-based Reinforcement Learning Agent for Autonomous Racing in Gran Turismocited by 15
  11. 2022Finite-Time Analysis of Natural Actor-Critic for POMDPscited by 13
  12. 2022Recurrent networks, hidden states and beliefs in partially observable environmentscited by 12
  13. 2023Informed POMDP: Leveraging Additional Information in Model-Based RLcited by 11
  14. 2023The Wasserstein Believer: Learning Belief Updates for Partially Observable Environments through Reliable Latent Space Modelscited by 9
  15. 2024Convergence of Gradient Descent for Recurrent Neural Networks: A Nonasymptotic Analysiscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2025Informed Asymmetric Actor-Critic: Leveraging Privileged Signals Beyond Full-State Accesscited by 2
  2. 2026Overcoming Scaling Limits in On-Policy Self-Distillation for LLM Reasoningcited by 1
  3. 2026Reinformed Dreamer: An Asymmetric World Model Efficiently Trained through Latent Guidancecited by 0
  4. 2026Learning Suffers More Than the Policy Class Under Partial Observability: A Closed-Form Analysiscited by 0
  5. 2026Shared Actors Need Not Share Critics: Effects of Value Mismatch in Parallel Reinforcement Learningcited by 0
  6. 2026Memory-State Critic for Asymmetric Actor-Critic with Application to Vision-Based Pursuit-Evasioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning
  2. 2025Quasi-Newton Compatible Actor-Critic for Deterministic Policies
  3. 2024Compatible Gradient Approximations for Actor-Critic Algorithms
  4. 2024Non-Asymptotic Analysis for Single-Loop (Natural) Actor-Critic with Compatible Function Approximation
  5. 2025Finite Time Analysis of Constrained Natural Critic-Actor Algorithm with Improved Sample Complexity
  6. 2025Finite-time Convergence Analysis of Actor-Critic with Evolving Reward
  7. 2025A Communication-Efficient Decentralized Actor-Critic Algorithm
  8. 2025Actor-Critic without Actor