# Robust Q-learning for mean-field control under Wasserstein uncertainty in common noise

arXiv: https://arxiv.org/abs/2606.20356 · HTML: https://mlsanity.com/paper/2606.20356/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 53 of its 91 references that are in the map.
- First posted on arXiv 18 Jun 2026.

## Builds on (most cited first, 30 of 53)

- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [Data-driven Distributionally Robust Optimization Using the Wasserstein Metric: Performance Guarantees and Tractable Reformulations](https://mlsanity.com/paper/1505.05116/) (2015)
- [Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism](https://mlsanity.com/paper/2103.12021/) (2021)
- [Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations](https://mlsanity.com/paper/2003.08938/) (2020)
- [Distributionally Robust Stochastic Optimization with Wasserstein Distance](https://mlsanity.com/paper/1604.02199/) (2016)
- [Breaking the Sample Size Barrier in Model-Based Reinforcement Learning with a Generative Model](https://mlsanity.com/paper/2005.12900/) (2020)
- [Finite-Time Analysis of Asynchronous Stochastic Approximation and Q-Learning](https://mlsanity.com/paper/2002.00260/) (2020)
- [Robust Reinforcement Learning using Offline Data](https://mlsanity.com/paper/2208.05129/) (2022)
- [Sample Complexity of Asynchronous Q-Learning: Sharper Analysis and Variance Reduction](https://mlsanity.com/paper/2006.03041/) (2020)
- [Settling the Sample Complexity of Model-Based Offline Reinforcement Learning](https://mlsanity.com/paper/2204.05275/) (2022)
- [Approximately Solving Mean Field Games via Entropy-Regularized Deep Reinforcement Learning](https://mlsanity.com/paper/2102.01585/) (2021)
- [Learning in Mean Field Games: A Survey](https://mlsanity.com/paper/2205.12944/) (2022)
- [Model-Free Mean-Field Reinforcement Learning: Mean-Field MDP and Mean-Field Q-Learning](https://mlsanity.com/paper/1910.12802/) (2019)
- [Reinforcement Learning under Model Mismatch](https://mlsanity.com/paper/1706.04711/) (2017)
- [Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coverage](https://mlsanity.com/paper/2305.09659/) (2023)
- [Mean-Field Controls with Q-learning for Cooperative MARL: Convergence and Complexity Analysis](https://mlsanity.com/paper/2002.04131/) (2020)
- [On the Convergence of Model Free Learning in Mean Field Games](https://mlsanity.com/paper/1907.02633/) (2019)
- [Q-Learning in Regularized Mean-field Games](https://mlsanity.com/paper/2003.12151/) (2020)
- [Scalable Deep Reinforcement Learning Algorithms for Mean Field Games](https://mlsanity.com/paper/2203.11973/) (2022)
- [Mean-field Markov decision processes with common noise and open-loop controls](https://mlsanity.com/paper/1912.07883/) (2019)
- [Recent Developments in Machine Learning Methods for Stochastic Control and Games](https://mlsanity.com/paper/2303.10257/) (2023)
- [Dynamic Programming Principles for Mean-Field Controls with Learning](https://mlsanity.com/paper/1911.07314/) (2019)
- [Numerical Methods for Mean Field Games and Mean Field Type Control](https://mlsanity.com/paper/2106.06231/) (2021)
- [Actor-Critic learning for mean-field control in continuous time](https://mlsanity.com/paper/2303.06993/) (2023)
- [Continuous-time q-learning for mean-field control problems](https://mlsanity.com/paper/2306.16208/) (2023)
- [Policy Gradient Algorithms for Robust MDPs with Non-Rectangular Uncertainty Sets](https://mlsanity.com/paper/2305.19004/) (2023)
- [Exploratory LQG Mean Field Games with Entropy Regularization](https://mlsanity.com/paper/2011.12946/) (2020)
- [Distributionally Robust Markov Decision Processes and their Connection to Risk Measures](https://mlsanity.com/paper/2007.13103/) (2020)
- [Unified continuous-time q-learning for mean-field game and mean-field control problems](https://mlsanity.com/paper/2407.04521/) (2024)

## Abstract (arXiv)

In this article, we present a robust Q-learning algorithm for discrete-time mean-field control problems under Wasserstein uncertainty in the common noise law. The algorithm combines a quantization-and-projection scheme with a Wasserstein dual reformulation on the common-noise space. We establish its convergence together with finite-time iteration bounds for both synchronous and asynchronous learning schemes. Numerical experiments on systemic risk and epidemic models compare the asynchronous implementation with an idealized Bellman iteration, illustrate the robustness-performance tradeoff under common-noise misspecification, and report the observed convergence behavior of the asynchronous Q-learning algorithm.