A Single Online Agent Can Efficiently Learn Mean Field Games
Not on the map. Why?
Abstract (arXiv)
Mean field games (MFGs) are a promising framework for modeling the behavior of large-population systems. However, solving MFGs can be challenging due to the coupling of forward population evolution and backward agent dynamics. Typically, obtaining mean field Nash equilibria (MFNE) involves an iterative approach where the forward and backward processes are solved alternately, known as fixed-point iteration (FPI). This method requires fully observed population propagation and agent dynamics over the entire spatial domain, which could be impractical in some real-world scenarios. To overcome this limitation, this paper introduces a novel online single-agent model-free learning scheme, which enables a single agent to learn MFNE using online samples, without prior knowledge of the state-action space, reward function, or transition dynamics. Specifically, the agent updates its policy through the value function (Q), while simultaneously evaluating the mean field state (M), using the same batch of observations. We develop two variants of this learning scheme: off-policy and on-policy QM iteration. We prove that they efficiently approximate FPI, and a sample complexity guarantee is provided. The efficacy of our methods is confirmed by numerical experiments.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 24 of its 41 references that are in the map. First posted on arXiv 5 May 2024.
Builds on
24 of 41 references.
References · 24
- 2018A Finite Time Analysis of Temporal Difference Learning With Linear Function Approximationcited by 229
- 2017On the Properties of the Softmax Function with Application in Game Theory and Reinforcement Learningcited by 142
- 2020Finite-Time Analysis of Asynchronous Stochastic Approximation and Q-Learningcited by 96
- 2021Is Q-Learning Minimax Optimal? A Tight Sample Complexity Analysiscited by 92
- 2019Finite-Sample Analysis for SARSA with Linear Function Approximationcited by 85
- 2021Approximately Solving Mean Field Games via Entropy-Regularized Deep Reinforcement Learningcited by 73
- 2020Fictitious Play for Mean Field Games: Continuous Time Analysis and Applicationscited by 62
- 2022Learning in Mean Field Games: A Surveycited by 61
- 2019Model-Free Mean-Field Reinforcement Learning: Mean-Field MDP and Mean-Field Q-Learningcited by 59
- 2020Entropy Regularization for Mean Field Games with Learningcited by 58
- 2020Unified Reinforcement Q-Learning for Mean Field Game and Control Problemscited by 49
- 2019On the Convergence of Model Free Learning in Mean Field Gamescited by 48
12 more
- 2020Q-Learning in Regularized Mean-field Gamescited by 42
- 2015Learning in Mean Field Games: the Fictitious Playcited by 38
- 2022Scalable Deep Reinforcement Learning Algorithms for Mean Field Gamescited by 38
- 2021Scaling up Mean Field Games with Online Mirror Descentcited by 32
- 2021Mean Field Games Flock! The Reinforcement Learning Waycited by 31
- 2024Finite-Time Analysis of On-Policy Heterogeneous Federated Reinforcement Learningcited by 31
- 2018Decentralised Learning in Systems with Many, Many Strategic Agentscited by 20
- 2022Oracle-free Reinforcement Learning in Mean-Field Games along a Single Sample Pathcited by 20
- 2022A Survey on Large-Population Systems and Scalable Multi-Agent Reinforcement Learningcited by 14
- 2021Decentralized Mean Field Gamescited by 12
- 2023Convergence of Multi-Scale Reinforcement Q-Learning Algorithms for Mean Field Game and Control Problemscited by 10
- 2020Multi-Agent Reinforcement Learning for Markov Routing Games: A New Modeling Paradigm For Dynamic Traffic Assignmentcited by 9
Built on by
Citing papers per year: 2024: 2.
2 papers.