mlsanityData to 7 Oct 2026

Fully Online Decentralized Learning in Stochastic Games with Unknown Independent Chains

arXiv 2610.01181 ↗First posted 1 Oct 2026cs.LG cs.GT cs.MAcited by 0

Not on the map. Why?

Abstract (arXiv)

We consider stochastic games with independent controlled chains and unknown transition kernels, where players observe only their local states and realized payoffs. We develop a fully online, decentralized, and uncoordinated mirror-descent algorithm that operates in the dual space of occupancy measures for approximating stationary Nash equilibrium (NE) policies. The algorithm uses a single transition/reward sample at every primitive time step, relies only on local information, and requires neither coverage of the joint state space nor synchronized episodes. Under uniform-ergodicity and finite-coverage assumptions, we show that, with high probability, the time-averaged fixed-comparator regret decays at the canonical O(T-1/2) rate, up to logarithmic factors and polynomial dependence on the game parameters. In particular, the complexity depends on the cover times of the individual local state spaces rather than the product state space, avoiding exponential dependence on the number of players and the sizes of the joint state and action spaces. The resulting finite-time regret bound further yields an approximate coarse-correlated-equilibrium guarantee, which is natural for arbitrary reward functions since computing a stationary ε-NE is PPAD-hard in this setting. Under an additional global variational-stability condition, we show that the same fully online algorithm converges asymptotically in the last iterate to a stationary ε-NE. Our results provide a fully online and scalable learning framework for stochastic games with unknown independent chains. The algorithm can also be viewed as a primal-dual framework for Markov games that exploits the independence and local structure of the players' controlled transition chains.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 22 of its 39 references that are in the map. First posted on arXiv 1 Oct 2026.

Builds on

22 of 39 references.

References · 22

  1. 2019Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmscited by 555
  2. 2021Independent Policy Gradient Methods for Competitive Reinforcement Learningcited by 97
  3. 2021Global Convergence of Multi-Agent Policy Gradient in Markov Potential Gamescited by 89
  4. 2019Learning Adversarial MDPs with Bandit Feedback and Unknown Transitioncited by 84
  5. 2016Learning in games with continuous action sets and unknown payoff functionscited by 78
  6. 2022Independent Policy Gradient for Large-Scale Markov Potential Games: Sharper Rates, Function Approximation, and Game-Agnostic Convergencecited by 54
  7. 2021Decentralized Q-Learning in Zero-sum Markov Gamescited by 53
  8. 2021Independent Natural Policy Gradient Always Converges in Markov Potential Gamescited by 34
  9. 2021Gradient play in stochastic games: stationary points, convergence, and sample complexitycited by 32
  10. 2020Fictitious play in zero-sum stochastic gamescited by 24
  11. 2022On the Global Convergence Rates of Decentralized Softmax Gradient Play in Markov Potential Gamescited by 22
  12. 2022Independent and Decentralized Learning in Markov Potential Gamescited by 19
10 more
  1. 2021Provably Efficient Policy Optimization for Two-Player Zero-Sum Markov Gamescited by 18
  2. 2023Provably Fast Convergence of Independent Natural Policy Gradient for Markov Potential Gamescited by 12
  3. 2024Independent Learning in Constrained Markov Potential Gamescited by 10
  4. 2022Provably Efficient Fictitious Play Policy Optimization for Zero-Sum Markov Games with Structured Transitionscited by 9
  5. 2019Large Scale Markov Decision Processes with Changing Rewardscited by 7
  6. 2016Stochastic Games for Smart Grid Energy Management with Prospect Prosumerscited by 6
  7. 2022Learning Stationary Nash Equilibrium Policies in n-Player Stochastic Games with Independent Chainscited by 6
  8. 2017Large Player games on Wireless Networkscited by 3
  9. 2026Independent Learning of Nash Equilibria in Partially Observable Markov Potential Games with Decoupled Dynamicscited by 1
  10. 2026High-Probability Nash Regret for Decentralized Learning in Markov α-Potential Games: Episodic and Fully Online Asynchronous Algorithms with Applications to Markov Congestion Gamescited by 1

Built on by

Not cited yet.