Learning Approximate Nash Equilibria in Cooperative Multi-Agent Reinforcement Learning via Mean-Field Subsampling
Not on the map. Why?
Abstract (arXiv)
Many large-scale platforms and networked control systems have a centralized decision maker interacting with a massive population of agents under strict observability constraints. Motivated by such applications, we study a cooperative Markov game with a global agent and n homogeneous local agents in a communication-constrained regime, where the global agent only observes a subset of k local agent states per time step. We propose an alternating learning framework (textttALTERNATING-MARL), where the global agent performs subsampled mean-field Q-learning against a fixed local policy, and local agents update by optimizing in an induced MDP. We prove that these approximate best-response dynamics converge to an widetildeO(1/√k)-approximate Nash Equilibrium, while separating the sample complexities between the joint state and action spaces. Finally, we validate our results in numerical simulations for multi-robot control.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 48 of its 90 references that are in the map. First posted on arXiv 4 Mar 2026.
Builds on
48 of 90 references.
References · 48
- 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
- 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
- 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
- 2016Safe, Multi-Agent, Reinforcement Learning for Autonomous Drivingcited by 301
- 2020Model-Based Reinforcement Learning with Value-Targeted Regressioncited by 233
- 2018Mean Field Multi-Agent Reinforcement Learningcited by 202
- 2021V-Learning -- A Simple, Efficient, Decentralized Algorithm for Multiagent RLcited by 111
- 2015Sample Complexity of Episodic Fixed-Horizon Reinforcement Learningcited by 105
- 2021Global Convergence of Multi-Agent Policy Gradient in Markov Potential Gamescited by 89
- 2019Variance-reduced Q-learning is minimax optimalcited by 72
- 2023Heterogeneous-Agent Reinforcement Learningcited by 67
- 2018Real-Time Bidding with Multi-Agent Reinforcement Learning in Display Advertisingcited by 65
36 more
- 2019Efficient Ridesharing Order Dispatching with Mean Field Multi-Agent Reinforcement Learningcited by 57
- 2021A Lyapunov Theory for Finite-Sample Guarantees of Asynchronous Q-Learning and TD-Learning Variantscited by 57
- 2017Variance Reduced Value Iteration and Faster Algorithms for Solving Markov Decision Processescited by 56
- 2022Independent Policy Gradient for Large-Scale Markov Potential Games: Sharper Rates, Function Approximation, and Game-Agnostic Convergencecited by 54
- 2020Mean-Field Controls with Q-learning for Cooperative MARL: Convergence and Complexity Analysiscited by 44
- 2021Sample-Efficient Learning of Stackelberg Equilibria in General-Sum Gamescited by 40
- 2021On the Approximation of Cooperative Heterogeneous Multi-Agent Reinforcement Learning (MARL) using Mean Field Control (MFC)cited by 31
- 2022Learning in Observable POMDPs, without Computationally Intractable Oraclescited by 30
- 2021Learning Graphon Mean Field Games and Approximate Nash Equilibriacited by 28
- 2021Mean-Field Multi-Agent Reinforcement Learning: A Decentralized Network Approachcited by 27
- 2023Concentration of Contractive Stochastic Approximation: Additive and Multiplicative Noisecited by 26
- 2021Almost Optimal Algorithms for Two-player Zero-Sum Linear Mixture Markov Gamescited by 25
- 2018Near-Optimal Time and Sample Complexities for Solving Discounted Markov Decision Process with a Generative Modelcited by 22
- 2019Scalable Reinforcement Learning for Multi-Agent Networked Systemscited by 22
- 2023Maximize to Explore: One Objective Function Fusing Estimation, Planning, and Explorationcited by 20
- 2021Finite-Sample Analysis of Off-Policy TD-Learning via Generalized Bellman Operatorscited by 17
- 2022Online Adaptive Policy Selection in Time-Varying Systems: No-Regret via Contractive Perturbationscited by 14
- 2021Permutation Invariant Policy Optimization for Mean-Field Multi-Agent Reinforcement Learning: A Principled Approachcited by 13
- 2021Decentralized Mean Field Gamescited by 12
- 2022No-Regret Learning in Dynamic Stackelberg Gamescited by 12
- 2014A Counter-Example to Karlin's Strong Conjecture for Fictitious Playcited by 10
- 2022Graphon Mean-Field Control for Cooperative Multi-Agent Reinforcement Learningcited by 10
- 2024Mean-Field Sampling for Cooperative Multi-Agent Reinforcement Learningcited by 9
- 2024Efficient Reinforcement Learning for Global Decision Making in the Presence of Local Agents at Scalecited by 8
- 2024Online Policy Optimization in Unknown Nonlinear Systemscited by 8
- 2024Peer-to-Peer Learning Dynamics of Wide Neural Networkscited by 8
- 2023Robust Stackelberg Equilibriacited by 7
- 2022Convergence and Price of Anarchy Guarantees of the Softmax Policy Gradient in Markov Potential Gamescited by 6
- 2024Truncated Variance Reduced Value Iterationcited by 6
- 2024A Local Information Aggregation based Multi-Agent Reinforcement Learning for Robot Swarm Dynamic Task Allocationcited by 6
- 2024Locally Interdependent Multi-Agent MDP: Theoretical Framework for Decentralized Agents with Dynamic Dependenciescited by 5
- 2024Linear Bellman Completeness Suffices for Efficient Online Reinforcement Learning with Few Actionscited by 4
- 2025The Structural Complexity of Matrix-Vector Multiplicationcited by 4
- 2025The Oversight Game: Learning to Cooperatively Balance an AI Agent's Safety and Autonomycited by 3
- 2026Unsupervised Decomposition and Recombination with Discriminator-Driven Diffusion Modelscited by 3
- 2026Graphon Mean-Field Subsampling for Cooperative Heterogeneous Multi-Agent Reinforcement Learningcited by 1
Built on by
Citing papers per year: 2026: 2.
2 papers.