mlsanityData to 14 Oct 2026

On the Statistical Efficiency of Mean-Field Reinforcement Learning with General Function Approximation

arXiv 2305.11283 ↗First posted 18 May 2023cs.LG cs.AI stat.MLcited by 14

Not on the map. Why?

Abstract (arXiv)

In this paper, we study the fundamental statistical efficiency of Reinforcement Learning in Mean-Field Control (MFC) and Mean-Field Game (MFG) with general model-based function approximation. We introduce a new concept called Mean-Field Model-Based Eluder Dimension (MF-MBED), which characterizes the inherent complexity of mean-field model classes. We show that a rich family of Mean-Field RL problems exhibits low MF-MBED. Additionally, we propose algorithms based on maximal likelihood estimation, which can return an ε-optimal policy for MFC or an ε-Nash Equilibrium policy for MFG. The overall sample complexity depends only polynomially on MF-MBED, which is potentially much lower than the size of state-action space. Compared with previous works, our results only require the minimal assumptions including realizability and Lipschitz continuity.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 49 of its 79 references that are in the map. First posted on arXiv 18 May 2023.

Builds on

49 of 79 references.

References · 49

  1. 2019Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmscited by 567
  2. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 520
  3. 2017Minimax Regret Bounds for Reinforcement Learningcited by 503
  4. 2018Is Q-learning Provably Efficient?cited by 487
  5. 2016Safe, Multi-Agent, Reinforcement Learning for Autonomous Drivingcited by 311
  6. 2016Contextual Decision Processes with Low Bellman Rank are PAC-Learnablecited by 292
  7. 2020Model-Based Reinforcement Learning with Value-Targeted Regressioncited by 234
  8. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 233
  9. 2021Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithmscited by 223
  10. 2021The Statistical Complexity of Interactive Decision Makingcited by 218
  11. 2018Mean Field Multi-Agent Reinforcement Learningcited by 207
  12. 2020Learning Near Optimal Policies with Low Inherent Bellman Errorcited by 181
37 more
  1. 2021Bilinear Classes: A Structural Framework for Provable Generalization in RLcited by 178
  2. 2021Representation Learning for Online and Offline RL in Low-rank MDPscited by 129
  3. 2014Model-based Reinforcement Learning and the Eluder Dimensioncited by 116
  4. 2021V-Learning -- A Simple, Efficient, Decentralized Algorithm for Multiagent RLcited by 114
  5. 2020Near-Optimal Reinforcement Learning with Self-Playcited by 109
  6. 2019Sample Complexity of Reinforcement Learning using Linearly Combined Model Ensemblescited by 100
  7. 2020Learning Zero-Sum Simultaneous-Move Markov Games Using Function Approximation and Correlated Equilibriumcited by 100
  8. 2022When Is Partially Observable Reinforcement Learning Not Scary?cited by 100
  9. 2022The Role of Coverage in Online Reinforcement Learningcited by 83
  10. 2021Model-free Representation Learning and Exploration in Low-rank MDPscited by 82
  11. 2021Approximately Solving Mean Field Games via Entropy-Regularized Deep Reinforcement Learningcited by 73
  12. 2020Fictitious Play for Mean Field Games: Continuous Time Analysis and Applicationscited by 62
  13. 2019Model-Free Mean-Field Reinforcement Learning: Mean-Field MDP and Mean-Field Q-Learningcited by 59
  14. 2020Entropy Regularization for Mean Field Games with Learningcited by 58
  15. 2021Concave Utility Reinforcement Learning: the Mean-Field Game Viewpointcited by 56
  16. 2020Unified Reinforcement Q-Learning for Mean Field Game and Control Problemscited by 49
  17. 2019On the Convergence of Model Free Learning in Mean Field Gamescited by 48
  18. 2021The Power of Exploiter: Provable Multi-Agent RL in Large State Spacescited by 48
  19. 2021Towards General Function Approximation in Zero-Sum Markov Gamescited by 43
  20. 2019Policy Optimization Provably Converges to Nash Equilibria in Zero-Sum Linear Quadratic Gamescited by 42
  21. 2020Q-Learning in Regularized Mean-field Gamescited by 42
  22. 2021Efficient Model-Based Multi-Agent Mean-Field Reinforcement Learningcited by 38
  23. 2021Scaling up Mean Field Games with Online Mirror Descentcited by 32
  24. 2016Markov-Nash Equilibria in Mean-Field Games with Discounted Costcited by 31
  25. 2022A General Framework for Sample-Efficient Function Approximation in Reinforcement Learningcited by 31
  26. 2023Breaking the Curse of Multiagency: Provably Efficient Decentralized Multi-Agent RL with Function Approximationcited by 30
  27. 2021Mean-Field Multi-Agent Reinforcement Learning: A Decentralized Network Approachcited by 28
  28. 2022Sample-Efficient Reinforcement Learning of Partially Observable Markov Gamescited by 27
  29. 2023Breaking the Curse of Multiagents in a Large State Space: RL in Markov Games with Independent Linear Function Approximationcited by 27
  30. 2022Policy Mirror Ascent for Efficient and Independent Learning in Mean Field Gamescited by 26
  31. 2021Almost Optimal Algorithms for Two-player Zero-Sum Linear Mixture Markov Gamescited by 25
  32. 2022Towards Deployment-Efficient Reinforcement Learning: Lower Bound and Optimalitycited by 24
  33. 2016Mean Field Game of Controls and An Application To Trade Crowdingcited by 23
  34. 2022A Self-Play Posterior Sampling Algorithm for Zero-Sum Markov Gamescited by 16
  35. 2021Reinforcement Learning for Mean Field Games, with Applications to Economicscited by 15
  36. 2022Representation Learning for General-sum Low-rank Markov Gamescited by 15
  37. 2022Eluder-based Regret for Stochastic Contextual MDPscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 8; 2025: 2; 2026: 2.

14 papers.

Sort:

By area:

  1. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 127
  2. 2024When is Mean-Field Reinforcement Learning Tractable and Relevant?cited by 8
  3. 2023Learning Decentralized Partially Observable Mean Field Control for Artificial Collective Behaviorcited by 7
  4. 2024MF-OML: Online Mean-Field Reinforcement Learning with Occupation Measures for Large Population Gamescited by 5
  5. 2024Stochastic Semi-Gradient Descent for Learning Mean Field Games with Population-Aware Function Approximationcited by 5
  6. 2024Learning to Steer Markovian Agents under Model Uncertaintycited by 2
  7. 2024Exploiting Approximate Symmetry for Efficient Multi-Agent Reinforcement Learningcited by 2
  8. 2024Scalable Offline Reinforcement Learning for Mean Field Gamescited by 1
  9. 2025Steering No-Regret Agents in MFGs under Model Uncertaintycited by 1
  10. 2025Policy Optimization for Continuous-time Linear-Quadratic Graphon Mean Field Gamescited by 1
  11. 2023Networked Communication for Decentralised Agents in Mean-Field Gamescited by 0
  12. 2024Last Iterate Convergence in Monotone Mean Field Gamescited by 0
2 more
  1. 2026Towards Scaling Reinforcement Learning to Massive Populations: Learning Mean-Field Representationscited by 0
  2. 2026Fair Policy Optimization in Major-Minor Weakly Coupled Markov Decision Processescited by 0