# Decentralized Bandit Learning of Markov Perfect Equilibria

arXiv: https://arxiv.org/abs/2610.14333 · HTML: https://mlsanity.com/paper/2610.14333/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 25 references that are in the map.
- First posted on arXiv 9 Oct 2026.

## Builds on (most cited first, 15 of 15)

- [Learning in games with continuous action sets and unknown payoff functions](https://mlsanity.com/paper/1608.07310/) (2016)
- [Decentralized Q-Learning in Zero-sum Markov Games](https://mlsanity.com/paper/2106.02748/) (2021)
- [Decentralized Q-Learning for Stochastic Teams and Games](https://mlsanity.com/paper/1506.07924/) (2015)
- [Bandit learning in concave N-person games](https://mlsanity.com/paper/1810.01925/) (2018)
- [Independent Learning in Stochastic Games](https://mlsanity.com/paper/2111.11743/) (2021)
- [Independent and Decentralized Learning in Markov Potential Games](https://mlsanity.com/paper/2205.14590/) (2022)
- [Doubly Optimal No-Regret Online Learning in Strongly Monotone Games with Bandit Feedback](https://mlsanity.com/paper/2112.02856/) (2021)
- [Decentralized Learning for Optimality in Stochastic Dynamic Teams and Games with Local Control and Global State Information](https://mlsanity.com/paper/1903.05812/) (2019)
- [Improved Rates for Derivative Free Gradient Play in Strongly Monotone Games](https://mlsanity.com/paper/2111.09456/) (2021)
- [Learning Generalized Nash Equilibria in a Class of Convex Games](https://mlsanity.com/paper/1703.04113/) (2017)
- [Unsynchronized Decentralized Q-Learning: Two Timescale Analysis By Persistence](https://mlsanity.com/paper/2308.03239/) (2023)
- [Decentralized Policy Gradient for Nash Equilibria Learning of General-sum Stochastic Games](https://mlsanity.com/paper/2210.07651/) (2022)
- [Markov Perfect Equilibria in Discrete Finite-Player and Mean-Field Games](https://mlsanity.com/paper/2507.04540/) (2025)
- [Iterative Schemes for Markov Perfect Equilibria](https://mlsanity.com/paper/2507.20898/) (2025)
- [Provably Convergent Actor-Critic for MARL through Risk-aversion](https://mlsanity.com/paper/2602.12386/) (2026)

## Abstract (arXiv)

We study decentralized bandit learning in discounted infinite-horizon, general-sum Markov games with continuous actions. Agents observe the public state, but only their own realized rewards, and use simultaneous perturbation stochastic approximation to update their actions. Under strong monotonicity of the negative running-reward gradients and sufficiently small time steps, we prove that the fully decentralized procedure converges almost surely to the unique stationary Markov perfect equilibrium. We obtain an O(n-1/2) convergence rate and present a natural extension to episodic finite-horizon learning.