# Markov-Nash Equilibria in Mean-Field Games with Discounted Cost

arXiv: https://arxiv.org/abs/1612.07878 · HTML: https://mlsanity.com/paper/1612.07878/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Dec 2016.

## Built on by (most cited first, 30 of 31)

- [Approximately Solving Mean Field Games via Entropy-Regularized Deep Reinforcement Learning](https://mlsanity.com/paper/2102.01585/) (2021)
- [Learning Graphon Mean Field Games and Approximate Nash Equilibria](https://mlsanity.com/paper/2112.01280/) (2021)
- [Policy Mirror Ascent for Efficient and Independent Learning in Mean Field Games](https://mlsanity.com/paper/2212.14449/) (2022)
- [Oracle-free Reinforcement Learning in Mean-Field Games along a Single Sample Path](https://mlsanity.com/paper/2208.11639/) (2022)
- [A Survey on Large-Population Systems and Scalable Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2209.03859/) (2022)
- [Learning Sparse Graphon Mean Field Games](https://mlsanity.com/paper/2209.03880/) (2022)
- [On the Statistical Efficiency of Mean-Field Reinforcement Learning with General Function Approximation](https://mlsanity.com/paper/2305.11283/) (2023)
- [Unified continuous-time q-learning for mean-field game and mean-field control problems](https://mlsanity.com/paper/2407.04521/) (2024)
- [Individual-Level Inverse Reinforcement Learning for Mean Field Games](https://mlsanity.com/paper/2202.06401/) (2022)
- [Adversarial Inverse Reinforcement Learning for Mean Field Games](https://mlsanity.com/paper/2104.14654/) (2021)
- [When is Mean-Field Reinforcement Learning Tractable and Relevant?](https://mlsanity.com/paper/2402.05757/) (2024)
- [Discrete-Time Mean Field Control with Environment States](https://mlsanity.com/paper/2104.14900/) (2021)
- [MF-OML: Online Mean-Field Reinforcement Learning with Occupation Measures for Large Population Games](https://mlsanity.com/paper/2405.00282/) (2024)
- [Major-Minor Mean Field Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2303.10665/) (2023)
- [Hypergraphon Mean Field Games](https://mlsanity.com/paper/2203.16223/) (2022)
- [Mean Field Games on Weighted and Directed Graphs via Colored Digraphons](https://mlsanity.com/paper/2209.03887/) (2022)
- [Population-size-Aware Policy Optimization for Mean-Field Games](https://mlsanity.com/paper/2302.03364/) (2023)
- [Learning Mean-Field Control for Delayed Information Load Balancing in Large Queuing Systems](https://mlsanity.com/paper/2208.04777/) (2022)
- [Learning Discrete-Time Major-Minor Mean Field Games](https://mlsanity.com/paper/2312.10787/) (2023)
- [Exploiting Approximate Symmetry for Efficient Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2408.15173/) (2024)
- [Reinforcement Learning for Finite Space Mean-Field Type Games](https://mlsanity.com/paper/2409.18152/) (2024)
- [MESOB: Balancing Equilibria & Social Optimality](https://mlsanity.com/paper/2307.07911/) (2023)
- [Mean-Field Reinforcement Learning without Synchrony](https://mlsanity.com/paper/2602.18026/) (2026)
- [Networked Communication for Decentralised Agents in Mean-Field Games](https://mlsanity.com/paper/2306.02766/) (2023)
- [Networked Communication for Mean-Field Games with Function Approximation and Empirical Mean-Field Estimation](https://mlsanity.com/paper/2408.11607/) (2024)
- [Last Iterate Convergence in Monotone Mean Field Games](https://mlsanity.com/paper/2410.05127/) (2024)
- [Bounded Rationality Equilibrium Learning in Mean Field Games](https://mlsanity.com/paper/2411.07099/) (2024)
- [Mean-Field Control on Sparse Graphs: From Local Limits to GNNs via Neighborhood Distributions](https://mlsanity.com/paper/2601.21477/) (2026)
- [Stationary Robust Mean-Field Games under Model Mismatches](https://mlsanity.com/paper/2606.22579/) (2026)
- [Towards Scaling Reinforcement Learning to Massive Populations: Learning Mean-Field Representations](https://mlsanity.com/paper/2609.02928/) (2026)

## Abstract (arXiv)

In this paper, we consider discrete-time dynamic games of the mean-field type with a finite number N of agents subject to an infinite-horizon discounted-cost optimality criterion. The state space of each agent is a locally compact Polish space. At each time, the agents are coupled through the empirical distribution of their states, which affects both the agents' individual costs and their state transition probabilities. We introduce a new solution concept of the Markov-Nash equilibrium, under which a policy is player-by-player optimal in the class of all Markov policies. Under mild assumptions, we demonstrate the existence of a mean-field equilibrium in the infinite-population limit N → ∞, and then show that the policy obtained from the mean-field equilibrium is approximately Markov-Nash when the number of agents N is sufficiently large.