# Entropy Regularization for Mean Field Games with Learning

arXiv: https://arxiv.org/abs/2010.00145 · HTML: https://mlsanity.com/paper/2010.00145/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Sep 2020.

## Built on by (most cited first, 30 of 58)

- [Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithms](https://mlsanity.com/paper/2111.11232/) (2021)
- [Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approach](https://mlsanity.com/paper/2108.06655/) (2021)
- [Scalable Deep Reinforcement Learning Algorithms for Mean Field Games](https://mlsanity.com/paper/2203.11973/) (2022)
- [Generalization in Mean Field Games by Learning Master Policies](https://mlsanity.com/paper/2109.09717/) (2021)
- [A mean-field games laboratory for generative modeling](https://mlsanity.com/paper/2304.13534/) (2023)
- [Policy Mirror Ascent for Efficient and Independent Learning in Mean Field Games](https://mlsanity.com/paper/2212.14449/) (2022)
- [Reinforcement learning for linear-convex models with jumps via stability analysis of feedback controls](https://mlsanity.com/paper/2104.09311/) (2021)
- [Actor-Critic learning for mean-field control in continuous time](https://mlsanity.com/paper/2303.06993/) (2023)
- [Continuous-time q-learning for mean-field control problems](https://mlsanity.com/paper/2306.16208/) (2023)
- [Exploration-exploitation trade-off for continuous-time episodic reinforcement learning with linear-convex models](https://mlsanity.com/paper/2112.10264/) (2021)
- [Robust Risk-Aware Reinforcement Learning](https://mlsanity.com/paper/2108.10403/) (2021)
- [A Survey on Large-Population Systems and Scalable Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2209.03859/) (2022)
- [Learning Sparse Graphon Mean Field Games](https://mlsanity.com/paper/2209.03880/) (2022)
- [On the Statistical Efficiency of Mean-Field Reinforcement Learning with General Function Approximation](https://mlsanity.com/paper/2305.11283/) (2023)
- [Fast Policy Learning for Linear Quadratic Control with Entropy Regularization](https://mlsanity.com/paper/2311.14168/) (2023)
- [Unified continuous-time q-learning for mean-field game and mean-field control problems](https://mlsanity.com/paper/2407.04521/) (2024)
- [Accuracy of Discretely Sampled Stochastic Policies in Continuous-time Reinforcement Learning](https://mlsanity.com/paper/2503.09981/) (2025)
- [Entropy annealing for policy mirror descent in continuous time and space](https://mlsanity.com/paper/2405.20250/) (2024)
- [A Policy Gradient Framework for Stochastic Optimal Control Problems with Global Convergence Guarantee](https://mlsanity.com/paper/2302.05816/) (2023)
- [Continuous-time Risk-sensitive Reinforcement Learning via Quadratic Variation Penalty](https://mlsanity.com/paper/2404.12598/) (2024)
- [Adversarial Inverse Reinforcement Learning for Mean Field Games](https://mlsanity.com/paper/2104.14654/) (2021)
- [Data-Driven Merton's Strategies via Policy Randomization](https://mlsanity.com/paper/2312.11797/) (2023)
- [Optimal scheduling of entropy regulariser for continuous-time linear-quadratic reinforcement learning](https://mlsanity.com/paper/2208.04466/) (2022)
- [Deep Reinforcement Learning for Infinite Horizon Mean Field Problems in Continuous Spaces](https://mlsanity.com/paper/2309.10953/) (2023)
- [Convergence Analysis for Entropy-Regularized Control Problems: A Probabilistic Approach](https://mlsanity.com/paper/2406.10959/) (2024)
- [Continuous-time q-learning for mean-field control with common noise, part-I: Theoretical foundations](https://mlsanity.com/paper/2604.27372/) (2026)
- [Finding Regularized Competitive Equilibria of Heterogeneous Agent Macroeconomic Models with Reinforcement Learning](https://mlsanity.com/paper/2303.04833/) (2023)
- [Continuous Q-Score Matching: Diffusion Guided Reinforcement Learning for Continuous-Time Control](https://mlsanity.com/paper/2510.17122/) (2025)
- [Model-free policy gradient for discrete-time mean-field control](https://mlsanity.com/paper/2601.11217/) (2026)
- [Continuous-time q-learning for mean-field control with common noise, part-II: q-learning algorithms](https://mlsanity.com/paper/2604.27378/) (2026)

## Abstract (arXiv)

Entropy regularization has been extensively adopted to improve the efficiency, the stability, and the convergence of algorithms in reinforcement learning. This paper analyzes both quantitatively and qualitatively the impact of entropy regularization for Mean Field Game (MFG) with learning in a finite time horizon. Our study provides a theoretical justification that entropy regularization yields time-dependent policies and, furthermore, helps stabilizing and accelerating convergence to the game equilibrium. In addition, this study leads to a policy-gradient algorithm for exploration in MFG. Under this algorithm, agents are able to learn the optimal exploration scheduling, with stable and fast convergence to the game equilibrium.