# Learning in Nonzero-Sum Stochastic Games with Potentials

arXiv: https://arxiv.org/abs/2103.09284 · HTML: https://mlsanity.com/paper/2103.09284/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Mar 2021.

## Built on by (most cited first, 26 of 26)

- [Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2109.11251/) (2021)
- [Global Convergence of Multi-Agent Policy Gradient in Markov Potential Games](https://mlsanity.com/paper/2106.01969/) (2021)
- [Heterogeneous-Agent Reinforcement Learning](https://mlsanity.com/paper/2304.09870/) (2023)
- [Independent Policy Gradient for Large-Scale Markov Potential Games: Sharper Rates, Function Approximation, and Game-Agnostic Convergence](https://mlsanity.com/paper/2202.04129/) (2022)
- [Settling the Variance of Multi-Agent Policy Gradients](https://mlsanity.com/paper/2108.08612/) (2021)
- [Independent Natural Policy Gradient Always Converges in Markov Potential Games](https://mlsanity.com/paper/2110.10614/) (2021)
- [On Improving Model-Free Algorithms for Decentralized Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2110.05707/) (2021)
- [Gradient play in stochastic games: stationary points, convergence, and sample complexity](https://mlsanity.com/paper/2106.00198/) (2021)
- [Offline Pre-trained Multi-Agent Decision Transformer: One Big Sequence Model Tackles All SMAC Tasks](https://mlsanity.com/paper/2112.02845/) (2021)
- [ESCHER: Eschewing Importance Sampling in Games by Computing a History Value Function to Estimate Regret](https://mlsanity.com/paper/2206.04122/) (2022)
- [Provably Fast Convergence of Independent Natural Policy Gradient for Markov Potential Games](https://mlsanity.com/paper/2310.09727/) (2023)
- [Policy Diagnosis via Measuring Role Diversity in Cooperative Multi-agent RL](https://mlsanity.com/paper/2207.05683/) (2022)
- [Self-Play PSRO: Toward Optimal Populations in Two-Player Zero-Sum Games](https://mlsanity.com/paper/2207.06541/) (2022)
- [Learning Stationary Nash Equilibrium Policies in n-Player Stochastic Games with Independent Chains](https://mlsanity.com/paper/2201.12224/) (2022)
- [Unsynchronized Decentralized Q-Learning: Two Timescale Analysis By Persistence](https://mlsanity.com/paper/2308.03239/) (2023)
- [On the Convergence of Fictitious Play: A Decomposition Approach](https://mlsanity.com/paper/2205.01469/) (2022)
- [Learning Distributed and Fair Policies for Network Load Balancing as Markov Potential Game](https://mlsanity.com/paper/2206.01451/) (2022)
- [Ego-centric Learning of Communicative World Models for Autonomous Driving](https://mlsanity.com/paper/2506.08149/) (2025)
- [Taming Multi-Agent Reinforcement Learning with Estimator Variance Reduction](https://mlsanity.com/paper/2209.01054/) (2022)
- [Scalable and Independent Learning of Nash Equilibrium Policies in n-Player Stochastic Games with Unknown Independent Chains](https://mlsanity.com/paper/2312.01587/) (2023)
- [Risk-Sensitive Multi-Agent Reinforcement Learning in Network Aggregative Markov Games](https://mlsanity.com/paper/2402.05906/) (2024)
- [Reinforcement Learning With Reward Machines in Stochastic Games](https://mlsanity.com/paper/2305.17372/) (2023)
- [Risk Sensitivity in Markov Games and Multi-Agent Reinforcement Learning: A Systematic Review](https://mlsanity.com/paper/2406.06041/) (2024)
- [A Scalable Game Theoretic Approach for Coordination of Multiple Dynamic Systems](https://mlsanity.com/paper/2409.11358/) (2024)
- [The Bounds of Algorithmic Collusion; Q-learning, Gradient Learning, and the Folk Theorem](https://mlsanity.com/paper/2411.12725/) (2024)
- [Vairiational Stochastic Games](https://mlsanity.com/paper/2503.06037/) (2025)

## Abstract (arXiv)

Multi-agent reinforcement learning (MARL) has become effective in tackling discrete cooperative game scenarios. However, MARL has yet to penetrate settings beyond those modelled by team and zero-sum games, confining it to a small subset of multi-agent systems. In this paper, we introduce a new generation of MARL learners that can handle nonzero-sum payoff structures and continuous settings. In particular, we study the MARL problem in a class of games known as stochastic potential games (SPGs) with continuous state-action spaces. Unlike cooperative games, in which all agents share a common reward, SPGs are capable of modelling real-world scenarios where agents seek to fulfil their individual goals. We prove theoretically our learning method, SPot-AC, enables independent agents to learn Nash equilibrium strategies in polynomial time. We demonstrate our framework tackles previously unsolvable tasks such as Coordination Navigation and large selfish routing games and that it outperforms the state of the art MARL baselines such as MADDPG and COMIX in such scenarios.