# Decentralized Learning for Optimality in Stochastic Dynamic Teams and Games with Local Control and Global State Information

arXiv: https://arxiv.org/abs/1903.05812 · HTML: https://mlsanity.com/paper/1903.05812/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Mar 2019.

## Built on by (most cited first, 13 of 13)

- [Gradient play in stochastic games: stationary points, convergence, and sample complexity](https://mlsanity.com/paper/2106.00198/) (2021)
- [Satisficing Paths and Independent Multi-Agent Reinforcement Learning in Stochastic Games](https://mlsanity.com/paper/2110.04638/) (2021)
- [Markov α-Potential Games](https://mlsanity.com/paper/2305.12553/) (2023)
- [Fully Decentralized Cooperative Multi-Agent Reinforcement Learning: A Survey](https://mlsanity.com/paper/2401.04934/) (2024)
- [Unsynchronized Decentralized Q-Learning: Two Timescale Analysis By Persistence](https://mlsanity.com/paper/2308.03239/) (2023)
- [Convergence of Decentralized Actor-Critic Algorithm in General-sum Markov Games](https://mlsanity.com/paper/2409.04613/) (2024)
- [Logit-Q Dynamics for Efficient Learning in Stochastic Teams](https://mlsanity.com/paper/2302.09806/) (2023)
- [Paths to Equilibrium in Games](https://mlsanity.com/paper/2403.18079/) (2024)
- [Robust optimal policies for team Markov games](https://mlsanity.com/paper/2105.07405/) (2021)
- [Decentralized Multi-Agent Reinforcement Learning for Continuous-Space Stochastic Games](https://mlsanity.com/paper/2303.13539/) (2023)
- [Policy Optimization and Multi-agent Reinforcement Learning for Mean-variance Team Stochastic Games](https://mlsanity.com/paper/2503.22779/) (2025)
- [Decentralized Optimal Equilibrium Learning Over Dynamic Networks](https://mlsanity.com/paper/2609.17601/) (2026)
- [Decentralized Bandit Learning of Markov Perfect Equilibria](https://mlsanity.com/paper/2610.14333/) (2026)

## Abstract (arXiv)

Stochastic dynamic teams and games are rich models for decentralized systems and challenging testing grounds for multi-agent learning. Previous work that guaranteed team optimality assumed stateless dynamics, or an explicit coordination mechanism, or joint-control sharing. In this paper, we present an algorithm with guarantees of convergence to team optimal policies in teams and common interest games. The algorithm is a two-timescale method that uses a variant of Q-learning on the finer timescale to perform policy evaluation while exploring the policy space on the coarser timescale. Agents following this algorithm are "independent learners": they use only local controls, local cost realizations, and global state information, without access to controls of other agents. The results presented here are the first, to our knowledge, to give formal guarantees of convergence to team optimality using independent learners in stochastic dynamic teams and common interest games.