mlsanityData to 7 Oct 2026

Adversarially Guided Actor-Critic

arXiv 2102.04376 ↗First posted 8 Feb 2021cs.LG cs.AI stat.MLcited by 47

Not on the map. Why?

Abstract (arXiv)

Despite definite success in deep reinforcement learning problems, actor-critic algorithms are still confronted with sample inefficiency in complex environments, particularly in tasks where efficient exploration is a bottleneck. These methods consider a policy (the actor) and a value function (the critic) whose respective losses are built using different motivations and approaches. This paper introduces a third protagonist: the adversary. While the adversary mimics the actor by minimizing the KL-divergence between their respective action distributions, the actor, in addition to learning to solve the task, tries to differentiate itself from the adversary predictions. This novel objective stimulates the actor to follow strategies that could not have been correctly predicted from previous trajectories, making its behavior innovative in tasks where the reward is extremely rare. Our experimental analysis shows that the resulting Adversarially Guided Actor-Critic (AGAC) algorithm leads to more exhaustive exploration. Notably, AGAC outperforms current state-of-the-art methods on a set of various hard-exploration and procedurally-generated tasks.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Builds on 45 of its 56 references that are in the map. First posted on arXiv 8 Feb 2021.

Builds on

45 of 56 references.

References · 45

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  4. 2015Continuous control with deep reinforcement learningcited by 3,279
  5. 2015Trust Region Policy Optimizationcited by 2,503
  6. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  7. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  8. 2016OpenAI Gymcited by 1,684
  9. 2016Generative Adversarial Imitation Learningcited by 1,576
  10. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  11. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  12. 2018Exploration by Random Network Distillationcited by 966
33 more
  1. 2016Adversarial Machine Learning at Scalecited by 934
  2. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  3. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  4. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  5. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  6. 2017Towards Principled Methods for Training Generative Adversarial Networkscited by 460
  7. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 457
  8. 2019A Theory of Regularized Markov Decision Processescited by 279
  9. 2017Noisy Networks for Explorationcited by 262
  10. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
  11. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 252
  12. 2019Go-Explore: a New Approach for Hard-Exploration Problemscited by 207
  13. 2016Sample Efficient Actor-Critic with Experience Replaycited by 185
  14. 2016ViZDoom: A Doom-based AI Research Platform for Visual Reinforcement Learningcited by 178
  15. 2018Understanding the impact of entropy on policy optimizationcited by 171
  16. 2018A Study on Overfitting in Deep Reinforcement Learningcited by 170
  17. 2018Self-Imitation Learningcited by 135
  18. 2020RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environmentscited by 135
  19. 2017Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximationcited by 119
  20. 2018Generalization and Regularization in DQNcited by 117
  21. 2019Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learningcited by 102
  22. 2019Generalization in Reinforcement Learning with Selective Noise Injection and Information Bottleneckcited by 101
  23. 2019Observational Overfitting in Reinforcement Learningcited by 90
  24. 2018A Dissection of Overfitting and Generalization in Continuous Reinforcement Learningcited by 86
  25. 2020Learning with AMIGo: Adversarially Motivated Intrinsic Goalscited by 86
  26. 2018Learning to Understand Goal Specifications by Modelling Rewardcited by 80
  27. 2018Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generationcited by 80
  28. 2020Munchausen Reinforcement Learningcited by 68
  29. 2015Distributional Smoothing with Virtual Adversarial Trainingcited by 61
  30. 2016Connecting Generative Adversarial Networks and Actor-Critic Methodscited by 49
  31. 2020Leverage the Average: an Analysis of KL Regularization in RLcited by 33
  32. 2020Diversity Actor-Critic: Sample-Aware Entropy Regularization for Sample-Efficient Explorationcited by 20
  33. 2020Self-Imitation Advantage Learningcited by 15

Built on by

Cited by year

Citing papers per year: 2021: 8; 2022: 14; 2023: 8; 2024: 9; 2025: 8.

47 papers.

Sort:

By area:

  1. 2021MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Researchcited by 96
  2. 2022Exploration via Elliptical Episodic Bonusescited by 51
  3. 2021Adversarial Intrinsic Motivation for Reinforcement Learningcited by 40
  4. 2021MADE: Exploration via Maximizing Deviation from Explored Regionscited by 40
  5. 2022Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimalitycited by 35
  6. 2023On the Importance of Exploration for Generalization in Reinforcement Learningcited by 30
  7. 2021Decoupled Reinforcement Learning to Stabilise Intrinsically-Motivated Explorationcited by 25
  8. 2023A Study of Global and Episodic Bonuses for Exploration in Contextual MDPscited by 19
  9. 2024NAVIX: Scaling MiniGrid Environments with JAXcited by 16
  10. 2023Diversifying AI: Towards Creative Chess with AlphaZerocited by 15
  11. 2025Episodic Novelty Through Temporal Distancecited by 15
  12. 2022Learning General World Models in a Handful of Reward-Free Deploymentscited by 12
35 more
  1. 2022LECO: Learnable Episodic Count for Task-Specific Intrinsic Rewardcited by 11
  2. 2022Exploring through Random Curiosity with General Value Functionscited by 10
  3. 2021Programmatic Reward Design by Examplecited by 9
  4. 2022A Hierarchical Bayesian Approach to Inverse Reinforcement Learning with Symbolic Reward Machinescited by 9
  5. 2022An Evaluation Study of Intrinsic Motivation Techniques applied to Reinforcement Learning over Hard Exploration Environmentscited by 8
  6. 2023Intrinsic Language-Guided Exploration for Complex Long-Horizon Robotic Manipulation Taskscited by 8
  7. 2025The impact of intrinsic rewards on exploration in Reinforcement Learningcited by 8
  8. 2021Don't Do What Doesn't Matter: Intrinsic Motivation with Action Usefulnesscited by 7
  9. 2023Improving Intrinsic Exploration by Creating Stationary Objectivescited by 7
  10. 2024Constrained Ensemble Exploration for Unsupervised Skill Discoverycited by 7
  11. 2021Explore and Control with Adversarial Surprisecited by 6
  12. 2024Settling Decentralized Multi-Agent Coordinated Exploration by Novelty Sharingcited by 6
  13. 2024Constrained Intrinsic Motivation for Reinforcement Learningcited by 6
  14. 2025CALM: Curiosity-Driven Auditing for Large Language Modelscited by 6
  15. 2022Curiosity in Hindsight: Intrinsic Exploration in Stochastic Environmentscited by 4
  16. 2022Assessing Quality-Diversity Neuro-Evolution Algorithms Performance in Hard Exploration Problemscited by 4
  17. 2022TD3 with Reverse KL Regularizer for Offline Reinforcement Learning from Mixed Datasetscited by 4
  18. 2022Towards Improving Exploration in Self-Imitation Learning using Intrinsic Motivationcited by 3
  19. 2023Toward Evaluating Robustness of Reinforcement Learning with Adversarial Policycited by 3
  20. 2023Go Beyond Imagination: Maximizing Episodic Reachability with World Modelscited by 3
  21. 2024Exploration by Learning Diverse Skills through Successor State Measurescited by 2
  22. 2022CIM: Constrained Intrinsic Motivation for Sparse-Reward Continuous Controlcited by 1
  23. 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environmentscited by 1
  24. 2024Joint Intrinsic Motivation for Coordinated Exploration in Multi-Agent Deep Reinforcement Learningcited by 1
  25. 2024Training on Irrelevant States Implies Data Augmentation: Generalization in Contextual MDPscited by 1
  26. 2025CAE: Repurposing the Critic as an Explorer in Deep Reinforcement Learningcited by 1
  27. 2025Beyond Noisy-TVs: Noise-Robust Exploration Via Learning Progress Monitoringcited by 1
  28. 2025Robust Agents in Open-Ended Worldscited by 1
  29. 2021More Efficient Exploration with Symbolic Priors on Action Sequence Equivalencescited by 0
  30. 2022q-Munchausen Reinforcement Learningcited by 0
  31. 2022Curiosity creates Diversity in Policy Searchcited by 0
  32. 2024Explore-Go: Leveraging Exploration for Generalisation in Deep Reinforcement Learningcited by 0
  33. 2024Fostering Intrinsic Motivation in Reinforcement Learning with Pretrained Foundation Modelscited by 0
  34. 2025CSAOT: Cooperative Multi-Agent System for Active Object Trackingcited by 0
  35. 2025KEA: Keeping Exploration Alive by Proactively Coordinating Exploration Strategiescited by 0