Independent RL for Cooperative-Competitive Agents: A Mean-Field Perspective
Not on the map. Why?
Abstract (arXiv)
We address in this paper Reinforcement Learning (RL) among agents that are grouped into teams such that there is cooperation within each team but general-sum (non-zero sum) competition across different teams. To develop an RL method that provably achieves a Nash equilibrium, we focus on a linear-quadratic structure. Moreover, to tackle the non-stationarity induced by multi-agent interactions in the finite population setting, we consider the case where the number of agents within each team is infinite, i.e., the mean-field setting. This results in a General-Sum LQ Mean-Field Type Game (GS-MFTG). We characterize the Nash equilibrium (NE) of the GS-MFTG, under a standard invertibility condition. This MFTG NE is then shown to be O(1/M)-NE for the finite population game where M is a lower bound on the number of agents in each team. These structural results motivate an algorithm called Multi-player Receding-horizon Natural Policy Gradient (MRNPG), where each team minimizes its cumulative cost independently in a receding-horizon manner. Despite the non-convexity of the problem, we establish that the resulting algorithm converges to a global NE through a novel problem decomposition into sub-problems using backward recursive discrete-time Hamilton-Jacobi-Isaacs (HJI) equations, in which independent natural policy gradient is shown to exhibit linear convergence under time-independent diagonal dominance. Numerical studies included corroborate the theoretical results.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 26 of its 50 references that are in the map. First posted on arXiv 17 Mar 2024.
Builds on
26 of 50 references.
References · 26
- 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
- 2019Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmscited by 555
- 2018Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulatorcited by 211
- 2018Mean Field Multi-Agent Reinforcement Learningcited by 202
- 2021V-Learning -- A Simple, Efficient, Decentralized Algorithm for Multiagent RLcited by 111
- 2018Derivative-Free Methods for Policy Optimization: Guarantees for Linear Quadratic Systemscited by 56
- 2019Model-Free Mean-Field Reinforcement Learning: Mean-Field MDP and Mean-Field Q-Learningcited by 56
- 2022Learning in Mean Field Games: A Surveycited by 56
- 2022Independent Policy Gradient for Large-Scale Markov Potential Games: Sharper Rates, Function Approximation, and Game-Agnostic Convergencecited by 54
- 2019On the Convergence of Model Free Learning in Mean Field Gamescited by 45
- 2020Mean-Field Controls with Q-learning for Cooperative MARL: Convergence and Complexity Analysiscited by 44
- 2020Unified Reinforcement Q-Learning for Mean Field Game and Control Problemscited by 42
14 more
- 2019Policy Optimization Provably Converges to Nash Equilibria in Zero-Sum Linear Quadratic Gamescited by 38
- 2021On Improving Model-Free Algorithms for Decentralized Multi-Agent Reinforcement Learningcited by 35
- 2019Distributed Reinforcement Learning for Decentralized Linear Quadratic Control: A Derivative-Free Policy Optimization Approachcited by 33
- 2022Scalable Deep Reinforcement Learning Algorithms for Mean Field Gamescited by 33
- 2019Linear-Quadratic Mean-Field Reinforcement Learning: Convergence of Policy Gradient Methodscited by 24
- 2016Mean Field Game of Controls and An Application To Trade Crowdingcited by 22
- 2021Policy Gradient Methods Find the Nash Equilibrium in N-player General-sum Linear-quadratic Gamescited by 19
- 2022Oracle-free Reinforcement Learning in Mean-Field Games along a Single Sample Pathcited by 16
- 2019Global Convergence of Policy Gradient for Sequential Zero-Sum Linear Quadratic Dynamic Gamescited by 14
- 2023Learning the Kalman Filter with Fine-Grained Sample Complexitycited by 9
- 2023Revisiting LQR Control from the Perspective of Receding-Horizon Policy Gradientcited by 9
- 2020Linear-Quadratic Zero-Sum Mean-Field Type Games: Optimality Conditions and Policy Optimizationcited by 5
- 2020Policy Optimization for Linear-Quadratic Zero-Sum Mean-Field Type Gamescited by 4
- 2021Adversarial Linear-Quadratic Mean-Field Games over Multigraphscited by 2
Built on by
Citing papers per year: 2024: 2.
2 papers.