mlsanityData to 14 Oct 2026

Multi-Agent Inverse Q-Learning from Demonstrations

arXiv 2503.04679 ↗First posted 6 Mar 2025cs.MA cs.AI cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

When reward functions are hand-designed, deep reinforcement learning algorithms often suffer from reward misspecification, causing them to learn suboptimal policies in terms of the intended task objectives. In the single-agent case, inverse reinforcement learning (IRL) techniques attempt to address this issue by inferring the reward function from expert demonstrations. However, in multi-agent problems, misalignment between the learned and true objectives is exacerbated due to increased environment non-stationarity and variance that scales with multiple agents. As such, in multi-agent general-sum games, multi-agent IRL algorithms have difficulty balancing cooperative and competitive objectives. To address these issues, we propose Multi-Agent Marginal Q-Learning from Demonstrations (MAMQL), a novel sample-efficient framework for multi-agent IRL. For each agent, MAMQL learns a critic marginalized over the other agents' policies, allowing for a well-motivated use of Boltzmann policies in the multi-agent context. We identify a connection between optimal marginalized critics and single-agent soft-Q IRL, allowing us to apply a direct, simple optimization criterion from the single-agent domain. Across our experiments on three different simulated domains, MAMQL significantly outperforms previous multi-agent methods in average reward, sample efficiency, and reward recovery by often more than 2-5x. We make our code available at https://sites.google.com/view/mamql .

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 28 of its 42 references that are in the map. First posted on arXiv 6 Mar 2025.

Builds on

28 of 42 references.

References · 28

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2016Generative Adversarial Imitation Learningcited by 1,597
  3. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,558
  4. 2019Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmscited by 567
  5. 2019On the Utility of Learning about Humans for Human-AI Coordinationcited by 349
  6. 2020"Other-Play" for Zero-Shot Coordinationcited by 198
  7. 2021IQ-Learn: Inverse soft-Q Learning for Imitationcited by 193
  8. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 176
  9. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 134
  10. 2020Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferencescited by 71
  11. 2021Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordinationcited by 70
  12. 2019Multi-Agent Adversarial Inverse Reinforcement Learningcited by 60
16 more
  1. 2021Deep Multi-agent Reinforcement Learning for Highway On-Ramp Merging in Mixed Trafficcited by 41
  2. 2019Social Attention for Autonomous Decision-Making in Dense Trafficcited by 39
  3. 2022Multi-agent Dynamic Algorithm Configurationcited by 36
  4. 2020When Humans Aren't Optimal: Robots that Collaborate with Risk-Aware Humanscited by 34
  5. 2022The Boltzmann Policy Distribution: Accounting for Systematic Suboptimality in Human Modelscited by 31
  6. 2021On the Critical Role of Conventions in Adaptive Human-AI Collaborationcited by 27
  7. 2021Evaluating the Robustness of Collaborative Agentscited by 25
  8. 2021Human irrationality: both bad and good for reward inferencecited by 25
  9. 2013Computational Rationalization: The Inverse Equilibrium Problemcited by 24
  10. 2020Reinforcement Learning based Control of Imitative Policies for Near-Accident Drivingcited by 23
  11. 2020Delay-Aware Multi-Agent Reinforcement Learning for Cooperative and Competitive Environmentscited by 22
  12. 2020SMART: Simultaneous Multi-Agent Recurrent Trajectory Predictioncited by 15
  13. 2021Inferring Objectives in Continuous Dynamic Games from Noise-Corrupted Partial State Observationscited by 13
  14. 2022Efficiently Computing Nash Equilibria in Adversarial Team Markov Gamescited by 7
  15. 2020Scalable Multi-Agent Inverse Reinforcement Learning via Actor-Attention-Criticcited by 6
  16. 2022Assisting Unknown Teammates in Unknown Tasks: Ad Hoc Teamwork under Partial Observabilitycited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Inverse Factorized Q-Learning for Cooperative Multi-agent Imitation Learning
  2. 2018Multi-agent Inverse Reinforcement Learning for Certain General-sum Stochastic Games
  3. 2026Generalize and Guide: Decomposing Rewards for Few-Shot Inverse Reinforcement Learning
  4. 2025Symmetry-Guided Multi-Agent Inverse Reinforcement Learning
  5. 2024On Feasible Rewards in Multi-Agent Inverse Reinforcement Learning
  6. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress
  7. 2018Competitive Multi-agent Inverse Reinforcement Learning with Sub-optimal Demonstrations
  8. 2021Multi-Agent Inverse Reinforcement Learning: Suboptimal Demonstrations and Alternative Solution Concepts