mlsanityData to 7 Oct 2026

Mean-Field Control based Approximation of Multi-Agent Reinforcement Learning in Presence of a Non-decomposable Shared Global State

arXiv 2301.06889 ↗First posted 13 Jan 2023cs.LG cs.AI cs.MAcited by 8

Not on the map. Why?

Abstract (arXiv)

Mean Field Control (MFC) is a powerful approximation tool to solve large-scale Multi-Agent Reinforcement Learning (MARL) problems. However, the success of MFC relies on the presumption that given the local states and actions of all the agents, the next (local) states of the agents evolve conditionally independent of each other. Here we demonstrate that even in a MARL setting where agents share a common global state in addition to their local states evolving conditionally independently (thus introducing a correlation between the state transition processes of individual agents), the MFC can still be applied as a good approximation tool. The global state is assumed to be non-decomposable i.e., it cannot be expressed as a collection of local states of the agents. We compute the approximation error as O(e) where e=frac1√Nleft[√|X| +√|U|right]. The size of the agent population is denoted by the term N, and |X|, |U| respectively indicate the sizes of (local) state and action spaces of individual agents. The approximation error is found to be independent of the size of the shared global state space. We further demonstrate that in a special case if the reward and state transition functions are independent of the action distribution of the population, then the error can be improved to e=frac√|X|√N. Finally, we devise a Natural Policy Gradient based algorithm that solves the MFC problem with O(ε-3) sample complexity and obtains a policy that is within O(maxe,ε) error of the optimal MARL policy for any ε>0.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 21 of its 34 references that are in the map. First posted on arXiv 13 Jan 2023.

Builds on

21 of 34 references.

References · 21

  1. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  2. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 697
  3. 2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 490
  4. 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 372
  5. 2020Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 197
  6. 2022An Improved Analysis of (Variance-Reduced) Policy Gradient and Natural Policy Gradient Methodscited by 63
  7. 2019Model-Free Mean-Field Reinforcement Learning: Mean-Field MDP and Mean-Field Q-Learningcited by 56
  8. 2019DeepPool: Distributed Model-free Algorithm for Ride-sharing using Deep Reinforcement Learningcited by 54
  9. 2020Scalable Multi-Agent Reinforcement Learning for Networked Systems with Average Rewardcited by 46
  10. 2019On the Convergence of Model Free Learning in Mean Field Gamescited by 45
  11. 2020Mean-Field Controls with Q-learning for Cooperative MARL: Convergence and Complexity Analysiscited by 44
  12. 2020Unified Reinforcement Q-Learning for Mean Field Game and Control Problemscited by 42
9 more
  1. 2021Efficient Model-Based Multi-Agent Mean-Field Reinforcement Learningcited by 38
  2. 2019Large-Scale Traffic Signal Control Using a Novel Multi-Agent Reinforcement Learningcited by 33
  3. 2021On the Approximation of Cooperative Heterogeneous Multi-Agent Reinforcement Learning (MARL) using Mean Field Control (MFC)cited by 31
  4. 2020Multi-Agent Reinforcement Learning in Stochastic Networked Systemscited by 30
  5. 2021Learning Graphon Mean Field Games and Approximate Nash Equilibriacited by 28
  6. 2019Mean-field Markov decision processes with common noise and open-loop controlscited by 24
  7. 2015Optimal resource allocation for competitive spreading processes on bilayer networkscited by 6
  8. 2022Can Mean Field Control (MFC) Approximate Cooperative Multi Agent Reinforcement Learning (MARL) with Non-Uniform Interaction?cited by 6
  9. 2022On the Near-Optimality of Local Policies in Large Cooperative Multi-Agent Reinforcement Learningcited by 4

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 5.

8 papers.

  1. 2023Improved Sample Complexity Analysis of Natural Policy Gradient Algorithm with General Parameterization for Infinite Horizon Discounted Reward Markov Decision Processescited by 24
  2. 2023Continuous-time q-learning for mean-field control problemscited by 17
  3. 2024Sample-Efficient Constrained Reinforcement Learning with General Parameterizationcited by 11
  4. 2024Learning General Parameterized Policies for Infinite Horizon Average Reward Constrained MDPs via Primal-Dual Policy Gradient Algorithmcited by 9
  5. 2024Unified continuous-time q-learning for mean-field game and mean-field control problemscited by 7
  6. 2023Major-Minor Mean Field Multi-Agent Reinforcement Learningcited by 4
  7. 2024Last-Iterate Convergence of General Parameterized Policies in Constrained MDPscited by 0
  8. 2024Evaluating the Impact of Multiple DER Aggregators on Wholesale Energy Markets: A Hybrid Mean Field Approachcited by 0