# Towards Model-Free Learning in Dynamic Population Games: An Application to Karma Economies

arXiv: https://arxiv.org/abs/2605.11042 · HTML: https://mlsanity.com/paper/2605.11042/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 46 references that are in the map.
- First posted on arXiv 11 May 2026.

## Builds on (most cited first, 28 of 28)

- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Mean Field Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1802.05438/) (2018)
- [Approximately Solving Mean Field Games via Entropy-Regularized Deep Reinforcement Learning](https://mlsanity.com/paper/2102.01585/) (2021)
- [Fictitious Play for Mean Field Games: Continuous Time Analysis and Applications](https://mlsanity.com/paper/2007.03458/) (2020)
- [Learning in Mean Field Games: A Survey](https://mlsanity.com/paper/2205.12944/) (2022)
- [Increasing the Action Gap: New Operators for Reinforcement Learning](https://mlsanity.com/paper/1512.04860/) (2015)
- [On the Convergence of Model Free Learning in Mean Field Games](https://mlsanity.com/paper/1907.02633/) (2019)
- [Learning in Mean Field Games: the Fictitious Play](https://mlsanity.com/paper/1507.06280/) (2015)
- [Scalable Deep Reinforcement Learning Algorithms for Mean Field Games](https://mlsanity.com/paper/2203.11973/) (2022)
- [Scaling up Mean Field Games with Online Mirror Descent](https://mlsanity.com/paper/2103.00623/) (2021)
- [Mean Field Games Flock! The Reinforcement Learning Way](https://mlsanity.com/paper/2105.07933/) (2021)
- [Policy Mirror Ascent for Efficient and Independent Learning in Mean Field Games](https://mlsanity.com/paper/2212.14449/) (2022)
- [A General Framework for Learning Mean-Field Games](https://mlsanity.com/paper/2003.06069/) (2020)
- [Mean Field Game of Controls and An Application To Trade Crowding](https://mlsanity.com/paper/1610.09904/) (2016)
- [On the Convergence and Sample Complexity Analysis of Deep Q-Networks with ε-Greedy Exploration](https://mlsanity.com/paper/2310.16173/) (2023)
- [Online Market Equilibrium with Application to Fair Division](https://mlsanity.com/paper/2103.12936/) (2021)
- [Policy iteration method for time-dependent Mean Field Games systems with non-separable Hamiltonians](https://mlsanity.com/paper/2110.02552/) (2021)
- [Population-aware Online Mirror Descent for Mean-Field Games by Deep Reinforcement Learning](https://mlsanity.com/paper/2403.03552/) (2024)
- [Today Me, Tomorrow Thee: Efficient Resource Allocation in Competitive Settings using Karma Games](https://mlsanity.com/paper/1907.09198/) (2019)
- [A self-contained karma economy for the dynamic allocation of common resources](https://mlsanity.com/paper/2207.00495/) (2022)
- [Convergence of Actor-Critic Learning for Mean Field Games and Mean Field Control in Continuous Spaces](https://mlsanity.com/paper/2511.06812/) (2025)
- [Learning optimal policies in potential Mean Field Games: Smoothed Policy Iteration algorithms](https://mlsanity.com/paper/2212.04791/) (2022)
- [Finite-Sample Convergence Bounds for Trust Region Policy Optimization in Mean-Field Games](https://mlsanity.com/paper/2505.22781/) (2025)
- [Dynamic Population Games: A Tractable Intersection of Mean-Field Games and Population Games](https://mlsanity.com/paper/2104.14662/) (2021)
- [CARMA: Fair and efficient bottleneck congestion management via non-tradable karma credits](https://mlsanity.com/paper/2208.07113/) (2022)
- [A Vision for Trustworthy, Fair, and Efficient Socio-Technical Control using Karma Economies](https://mlsanity.com/paper/2506.17115/) (2025)
- [Towards Fair and Efficient allocation of Mobility-on-Demand resources through a Karma Economy](https://mlsanity.com/paper/2511.07225/) (2025)
- [Bench-MFG: A Benchmark Suite for Learning in Stationary Mean Field Games](https://mlsanity.com/paper/2602.12517/) (2026)

## Abstract (arXiv)

Dynamic Population Games (DPGs) provide a tractable framework for modeling strategic interactions in large populations of self-interested agents, and have been successfully applied to the design of Karma economies, a class of fair non-monetary resource allocation mechanisms. Despite their appealing theoretical properties, existing computational tools for DPGs assume full knowledge of the game model and operate in a centralized fashion, limiting their applicability in realistic settings where agents have access only to their own private experience. This paper takes a step towards addressing this gap by studying model-free equilibrium learning in Karma DPGs. First, we analyze the setting in which a novel agent joins a Karma DPG already at its Stationary Nash Equilibrium (SNE) and learns a policy via Deep Q-Networks (DQN) without knowledge of the game model. Leveraging recent convergence results for DQN, we establish a suboptimality bound consisting of a DQN approximation error of order O(1/√Ns) and a mean field perturbation error of order O(1/N), where Ns is the replay buffer size and N is the population size. Second, we consider the challenging problem of learning the SNE from scratch. We show empirically that combining deep RL with fictitious play and smoothed policy iteration allows agents to converge, in a model-free fashion, to a configuration close to the centrally computed SNE. Together, these contributions support the vision of Karma economies as practical tools for fair resource allocation.