# Recurrent Structural Policy Gradient for Partially Observable Mean Field Games

arXiv: https://arxiv.org/abs/2602.20141 · HTML: https://mlsanity.com/paper/2602.20141/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods.
- Builds on 35 of its 52 references that are in the map.
- First posted on arXiv 23 Feb 2026.

## Placed in

- Deep reinforcement learning methods / [RL for games and combinatorial optimization](https://mlsanity.com/deeprl/rl-for-games-and-combinatorial-optimization/)

## Builds on (most cited first, 30 of 35)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?](https://mlsanity.com/paper/2011.09533/) (2020)
- [OpenSpiel: A Framework for Reinforcement Learning in Games](https://mlsanity.com/paper/1908.09453/) (2019)
- [Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPs](https://mlsanity.com/paper/2110.05038/) (2021)
- [Discovered Policy Optimisation](https://mlsanity.com/paper/2210.05639/) (2022)
- [Approximately Solving Mean Field Games via Entropy-Regularized Deep Reinforcement Learning](https://mlsanity.com/paper/2102.01585/) (2021)
- [JaxMARL: Multi-Agent RL Environments and Algorithms in JAX](https://mlsanity.com/paper/2311.10090/) (2023)
- [Fictitious Play for Mean Field Games: Continuous Time Analysis and Applications](https://mlsanity.com/paper/2007.03458/) (2020)
- [Learning in Mean Field Games: A Survey](https://mlsanity.com/paper/2205.12944/) (2022)
- [On the Convergence of Model Free Learning in Mean Field Games](https://mlsanity.com/paper/1907.02633/) (2019)
- [Learning in Mean Field Games: the Fictitious Play](https://mlsanity.com/paper/1507.06280/) (2015)
- [Scalable Deep Reinforcement Learning Algorithms for Mean Field Games](https://mlsanity.com/paper/2203.11973/) (2022)
- [Scaling up Mean Field Games with Online Mirror Descent](https://mlsanity.com/paper/2103.00623/) (2021)
- [Mean Field Games Flock! The Reinforcement Learning Way](https://mlsanity.com/paper/2105.07933/) (2021)
- [Generalization in Mean Field Games by Learning Master Policies](https://mlsanity.com/paper/2109.09717/) (2021)
- [A General Framework for Learning Mean-Field Games](https://mlsanity.com/paper/2003.06069/) (2020)
- [Recent Developments in Machine Learning Methods for Stochastic Control and Games](https://mlsanity.com/paper/2303.10257/) (2023)
- [Mean Field Game of Controls and An Application To Trade Crowding](https://mlsanity.com/paper/1610.09904/) (2016)
- [Numerical Methods for Mean Field Games and Mean Field Type Control](https://mlsanity.com/paper/2106.06231/) (2021)
- [Deep Learning for Mean Field Games and Mean Field Control with Applications to Finance](https://mlsanity.com/paper/2107.04568/) (2021)
- [Partially Observable Mean Field Reinforcement Learning](https://mlsanity.com/paper/2012.15791/) (2020)
- [A Survey on Large-Population Systems and Scalable Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2209.03859/) (2022)
- [DeepHAM: A Global Solution Method for Heterogeneous Agent Models with Aggregate Shocks](https://mlsanity.com/paper/2112.14377/) (2021)
- [Approximate Nash Equilibria in Partially Observed Stochastic Games with Mean-Field Interactions](https://mlsanity.com/paper/1705.02036/) (2017)
- [Solving N-player dynamic routing games with congestion: a mean field approach](https://mlsanity.com/paper/2110.11943/) (2021)
- [MFGLib: A Library for Mean-Field Games](https://mlsanity.com/paper/2304.08630/) (2023)
- [Contact rate epidemic control of COVID-19: an equilibrium view](https://mlsanity.com/paper/2004.08221/) (2020)
- [Policy iteration method for time-dependent Mean Field Games systems with non-separable Hamiltonians](https://mlsanity.com/paper/2110.02552/) (2021)
- [Regularization of the policy updates for stabilizing Mean Field Games](https://mlsanity.com/paper/2304.01547/) (2023)
- [Population-aware Online Mirror Descent for Mean-Field Games by Deep Reinforcement Learning](https://mlsanity.com/paper/2403.03552/) (2024)

## Abstract (arXiv)

Mean Field Games (MFGs) provide a principled framework for modelling interactions in large population systems. However, algorithmic progress has been limited since model-free methods are high variance and exact methods scale poorly. Recent Hybrid Structural Methods (HSMs) reduce variance while maintaining tractability by leveraging low-dimensional individual state and action spaces and known transition dynamics to compute the exact expected return conditioned on Monte Carlo rollouts of common noise. However, HSMs have not been extended to partially observable settings. We propose Recurrent Structural Policy Gradient (RSPG), the first history-aware HSM for MFGs with public partial information. RSPG achieves an order-of-magnitude faster convergence than model-free RL methods while learning history-aware behaviour, unlike current HSMs. To facilitate research into MFGs, we also introduce MFAX, our JAX-based framework for MFGs that supports both analytic and sample-based mean-field updates. MFAX and usage examples can be found at https://clarisse-wibault.github.io/rspg/.