mlsanityData to 7 Oct 2026

Concave Utility Reinforcement Learning: the Mean-Field Game Viewpoint

arXiv 2106.03787 ↗First posted 7 Jun 2021cs.LG cs.MAcited by 54

Not on the map. Why?

Abstract (arXiv)

Concave Utility Reinforcement Learning (CURL) extends RL from linear to concave utilities in the occupancy measure induced by the agent's policy. This encompasses not only RL but also imitation learning and exploration, among others. Yet, this more general paradigm invalidates the classical Bellman equations, and calls for new algorithms. Mean-field Games (MFGs) are a continuous approximation of many-agent RL. They consider the limit case of a continuous distribution of identical agents, anonymous with symmetric interests, and reduce the problem to the study of a single representative agent in interaction with the full population. Our core contribution consists in showing that CURL is a subclass of MFGs. We think this important to bridge together both communities. It also allows to shed light on aspects of both fields: we show the equivalence between concavity in CURL and monotonicity in the associated MFG, between optimality conditions in CURL and Nash equilibrium in MFG, or that Fictitious Play (FP) for this class of MFGs is simply Frank-Wolfe, bringing the first convergence rate for discrete-time FP for MFGs. We also experimentally demonstrate that, using algorithms recently introduced for solving MFGs, we can address the CURL problem more efficiently.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). Builds on 28 of its 44 references that are in the map. First posted on arXiv 7 Jun 2021.

Builds on

28 of 44 references.

References · 28

  1. 2015Trust Region Policy Optimizationcited by 2,503
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  3. 2019Normalizing Flows for Probabilistic Modeling and Inferencecited by 1,315
  4. 2014Convex Optimization: Algorithms and Complexitycited by 569
  5. 2018Maximum a Posteriori Policy Optimisationcited by 318
  6. 2019A Theory of Regularized Markov Decision Processescited by 279
  7. 2018Provably Efficient Maximum Entropy Explorationcited by 240
  8. 2018Mean Field Multi-Agent Reinforcement Learningcited by 202
  9. 2019A Divergence Minimization Perspective on Imitation Learning Methodscited by 188
  10. 2019Efficient Exploration via State Marginal Matchingcited by 186
  11. 2020Variational Policy Gradient Method for Reinforcement Learning with General Utilitiescited by 108
  12. 2021Reward is enough for convex MDPscited by 82
16 more
  1. 2020Munchausen Reinforcement Learningcited by 68
  2. 2018A Mean-Field Optimal Control Formulation of Deep Learningcited by 67
  3. 2021Approximately Solving Mean Field Games via Entropy-Regularized Deep Reinforcement Learningcited by 65
  4. 2020Fictitious Play for Mean Field Games: Continuous Time Analysis and Applicationscited by 57
  5. 2019On the Convergence of Model Free Learning in Mean Field Gamescited by 45
  6. 2020Q-Learning in Regularized Mean-field Gamescited by 40
  7. 2015Learning in Mean Field Games: the Fictitious Playcited by 37
  8. 2021Scaling up Mean Field Games with Online Mirror Descentcited by 29
  9. 2018Decentralised Learning in Systems with Many, Many Strategic Agentscited by 19
  10. 2016Frank-Wolfe Algorithms for Saddle Point Problemscited by 18
  11. 2020Stable Policy Optimization via Off-Policy Divergence Regularizationcited by 16
  12. 2019Reinforcement Learning for Joint Optimization of Multiple Rewardscited by 15
  13. 2017Learning Deep Mean Field Games for Modeling Large Population Behaviorcited by 8
  14. 2020Connecting GANs, MFGs, and OTcited by 8
  15. 2020Provable Fictitious Play for General Mean-Field Gamescited by 7
  16. 2017Numerical Methods for Mean-Field-Type Optimal Control Problemscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 10; 2023: 11; 2024: 13; 2025: 6; 2026: 11.

54 papers.

Sort:

By area:

  1. 2021Efficient Model-Based Multi-Agent Mean-Field Reinforcement Learningcited by 38
  2. 2022Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimalitycited by 35
  3. 2022Scalable Deep Reinforcement Learning Algorithms for Mean Field Gamescited by 33
  4. 2022Challenging Common Assumptions in Convex Reinforcement Learningcited by 29
  5. 2021Generalization in Mean Field Games by Learning Master Policiescited by 28
  6. 2023Reinforcement Learning with General Utilities: Simpler Variance Reduction and Large State-Action Spacecited by 18
  7. 2023Scalable Primal-Dual Actor-Critic Method for Safe Multi-Agent RL with General Utilitiescited by 15
  8. 2023Diversifying AI: Towards Creative Chess with AlphaZerocited by 15
  9. 2024Global Reinforcement Learning: Beyond Linear and Convex Rewards via Submodular Semi-gradient Methodscited by 15
  10. 2023On the Statistical Efficiency of Mean-Field Reinforcement Learning with General Function Approximationcited by 13
  11. 2023On Imitation in Mean-field Gamescited by 10
  12. 2024On the Limitations of Markovian Rewards to Express Multi-Objective, Risk-Sensitive, and Modal Taskscited by 9
38 more
  1. 2023Inverse Reinforcement Learning with the Average Reward Criterioncited by 8
  2. 2023Active Coverage for PAC Reinforcement Learningcited by 8
  3. 2023Learning Regularized Monotone Graphon Mean-Field Gamescited by 8
  4. 2022Lazy-MDPs: Towards Interpretable Reinforcement Learning by Learning When to Actcited by 7
  5. 2022Improved Policy Optimization for Online Imitation Learningcited by 6
  6. 2022Policy Gradient for Reinforcement Learning with General Utilitiescited by 6
  7. 2024The Limits of Pure Exploration in POMDPs: When the Observation Entropy is Enoughcited by 6
  8. 2024Geometric Active Exploration in Markov Decision Processes: the Benefit of Abstractioncited by 6
  9. 2024On the Global Optimality of Policy Gradient Methods in General Utility Reinforcement Learningcited by 6
  10. 2023Cancellation-Free Regret Bounds for Lagrangian Approaches in Constrained Markov Decision Processescited by 5
  11. 2024Population-aware Online Mirror Descent for Mean-Field Games by Deep Reinforcement Learningcited by 5
  12. 2024MF-OML: Online Mean-Field Reinforcement Learning with Occupation Measures for Large Population Gamescited by 5
  13. 2022Learning Correlated Equilibria in Mean-Field Gamescited by 4
  14. 2024MetaCURL: Non-stationary Concave Utility Reinforcement Learningcited by 4
  15. 2025Population-aware Online Mirror Descent for Mean-Field Games with Common Noise by Deep Reinforcement Learningcited by 4
  16. 2021Exploration noise for learning linear-quadratic mean field gamescited by 3
  17. 2024The Number of Trials Matters in Infinite-Horizon General-Utility Markov Decision Processescited by 3
  18. 2024Convex Markov Games: A New Frontier for Multi-Agent Reinforcement Learningcited by 3
  19. 2022Developing, Evaluating and Scaling Learning Agents in Multi-Agent Environmentscited by 2
  20. 2022Contextual bandits with concave rewards, and an application to fair rankingcited by 2
  21. 2024Inverse Concave-Utility Reinforcement Learning is Inverse Game Theorycited by 2
  22. 2025Finite-Sample Convergence Bounds for Trust Region Policy Optimization in Mean-Field Gamescited by 2
  23. 2025Solving Continuous Mean Field Games: Deep Reinforcement Learning for Non-Stationary Dynamicscited by 2
  24. 2026Soft Forward-Backward Representations for Zero-shot Reinforcement Learning with General Utilitiescited by 2
  25. 2026Breaking the Bias Barrier in Concave Multi-Objective Reinforcement Learningcited by 2
  26. 2024A Dual Perspective of Reinforcement Learning for Imposing Policy Constraintscited by 1
  27. 2025Solving General-Utility Markov Decision Processes in the Single-Trial Regime with Online Planningcited by 1
  28. 2025The Geometry of Nonlinear Reinforcement Learningcited by 1
  29. 2026Convex Markov Games and Beyond: New Proof of Existence, Characterization and Learning Algorithms for Nash Equilibriacited by 1
  30. 2026Bench-MFG: A Benchmark Suite for Learning in Stationary Mean Field Gamescited by 1
  31. 2026A Single Deep Preference-Conditioned Policy for Learning Pareto Coverage Setscited by 1
  32. 2026Active Inference as a Convex Markov Decision Processcited by 1
  33. 2026Robust General Utility for Reinforcement Learningcited by 1
  34. 2022On the importance of data collection for training general goal-reaching policiescited by 0
  35. 2023Reimagining Demand-Side Management with Mean Field Learningcited by 0
  36. 2023Networked Communication for Decentralised Agents in Mean-Field Gamescited by 0
  37. 2024Last Iterate Convergence in Monotone Mean Field Gamescited by 0
  38. 2025Online Episodic Convex Reinforcement Learningcited by 0

50 most cited of 54 All 54 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Efficient Model-Based Concave Utility Reinforcement Learning through Greedy Mirror Descent
  2. 2022Individual-Level Inverse Reinforcement Learning for Mean Field Games
  3. 2026Population-Aware Imitation Learning in Mean-field Games with Common Noise
  4. 2021Adversarial Inverse Reinforcement Learning for Mean Field Games
  5. 2024Scalable Offline Reinforcement Learning for Mean Field Games
  6. 2026Matching Multiple Experts: On the Exploitability of Multi-Agent Imitation Learning
  7. 2024Maximum Causal Entropy IRL in Mean-Field Games and GNEP Framework for Forward RL
  8. 2026NashDreamer: Model-Based Reinforcement Learning for Zero-Sum Imperfect-Information Games