mlsanityData to 7 Oct 2026

Multi-Agent Advisor Q-Learning

arXiv 2111.00345 ↗First posted 26 Oct 2021cs.AI cs.MAcited by 4

Not on the map. Why?

Abstract (arXiv)

In the last decade, there have been significant advances in multi-agent reinforcement learning (MARL) but there are still numerous challenges, such as high sample complexity and slow convergence to stable policies, that need to be overcome before wide-spread deployment is possible. However, many real-world environments already, in practice, deploy sub-optimal or heuristic approaches for generating policies. An interesting question that arises is how to best use such approaches as advisors to help improve reinforcement learning in multi-agent domains. In this paper, we provide a principled framework for incorporating action recommendations from online sub-optimal advisors in multi-agent settings. We describe the problem of ADvising Multiple Intelligent Reinforcement Agents (ADMIRAL) in nonrestrictive general-sum stochastic game environments and present two novel Q-learning based algorithms: ADMIRAL - Decision Making (ADMIRAL-DM) and ADMIRAL - Advisor Evaluation (ADMIRAL-AE), which allow us to improve learning by appropriately incorporating advice from an advisor (ADMIRAL-DM), and evaluate the effectiveness of an advisor (ADMIRAL-AE). We analyze the algorithms theoretically and provide fixed-point guarantees regarding their learning in general-sum stochastic games. Furthermore, extensive experiments illustrate that these algorithms: can be used in a variety of environments, have performances that compare favourably to other related baselines, can scale to large state-action spaces, and are robust to poor advice from advisors.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 47 of its 136 references that are in the map. First posted on arXiv 26 Oct 2021.

Builds on

47 of 136 references.

References · 47

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  4. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  5. 2016Generative Adversarial Imitation Learningcited by 1,576
  6. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  7. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
  8. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  9. 2019Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmscited by 555
  10. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  11. 2017End-to-end Driving via Conditional Imitation Learningcited by 479
  12. 2019Exploring the Limitations of Behavior Cloning for Autonomous Drivingcited by 343
35 more
  1. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  2. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  3. 2016Cooperative Inverse Reinforcement Learningcited by 298
  4. 2020PettingZoo: Gym for Multi-Agent Reinforcement Learningcited by 209
  5. 2017DART: Noise Injection for Robust Imitation Learningcited by 187
  6. 2015Maximum Entropy Deep Inverse Reinforcement Learningcited by 173
  7. 2018A Survey and Critique of Multiagent Deep Reinforcement Learningcited by 173
  8. 2016A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Modelscited by 167
  9. 2018Reinforcement and Imitation Learning for Diverse Visuomotor Skillscited by 124
  10. 2018Multi-Agent Generative Adversarial Imitation Learningcited by 99
  11. 2016Opponent Modeling in Deep Reinforcement Learningcited by 94
  12. 2017Imitating Driver Behavior with Generative Adversarial Networkscited by 92
  13. 2018Reinforcement Learning from Imperfect Demonstrationscited by 74
  14. 2017Coordinated Multi-Agent Imitation Learningcited by 70
  15. 2018Modeling Others using Oneself in Multi-Agent Reinforcement Learningcited by 64
  16. 2020A review of machine learning applications in wildfire science and managementcited by 64
  17. 2019Multi-Agent Adversarial Inverse Reinforcement Learningcited by 60
  18. 2018Learning to Teach in Cooperative Multiagent Reinforcement Learningcited by 45
  19. 2019Reinforcement Learning for Market Making in a Multi-agent Dealer Marketcited by 35
  20. 2016Learning a Driving Simulatorcited by 33
  21. 2018Pommerman: A Multi-Agent Playgroundcited by 30
  22. 2020Learning Guidance Rewards with Trajectory-space Smoothingcited by 30
  23. 2019Reinforcement Learning from Imperfect Demonstrations under Soft Expert Guidancecited by 25
  24. 2013Computational Rationalization: The Inverse Equilibrium Problemcited by 24
  25. 2017A Deep Policy Inference Q-Network for Multi-Agent Systemscited by 24
  26. 2014Multi-agent Inverse Reinforcement Learning for Two-person Zero-sum Gamescited by 17
  27. 2019Integrating Behavior Cloning and Reinforcement Learning for Improved Performance in Dense and Sparse Reward Environmentscited by 17
  28. 2020A Conceptual Framework for Externally-influenced Agents: An Assisted Reinforcement Learning Reviewcited by 15
  29. 2020Expert-Supervised Reinforcement Learning for Offline Policy Learning and Evaluationcited by 14
  30. 2018Competitive Multi-agent Inverse Reinforcement Learning with Sub-optimal Demonstrationscited by 11
  31. 2016Theoretically-Grounded Policy Advice from Multiple Teachers in Reinforcement Learning Settings with Applications to Negative Transfercited by 5
  32. 2019On Hard Exploration for Reinforcement Learning: a Case Study in Pommermancited by 4
  33. 2021Robust Maximum Entropy Behavior Cloningcited by 4
  34. 2019Accelerating Training in Pommerman with Imitation and Reinforcement Learningcited by 3
  35. 2021Shaping Advice in Deep Multi-Agent Reinforcement Learningcited by 3

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 2; 2024: 0; 2025: 1.

4 papers.

  1. 2022Explainable Action Advising for Multi-Agent Reinforcement Learningcited by 10
  2. 2023Expert-Free Online Transfer Learning in Multi-Agent Reinforcement Learningcited by 1
  3. 2023Learning from Multiple Independent Advisors in Multi-agent Reinforcement Learningcited by 0
  4. 2025Expert-Free Online Transfer Learning in Multi-Agent Reinforcement Learningcited by 0