# Deterministic limit of temporal difference reinforcement learning for stochastic games

arXiv: https://arxiv.org/abs/1809.07225 · HTML: https://mlsanity.com/paper/1809.07225/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Sep 2018.

## Built on by (most cited first, 5 of 5)

- [Modeling the effects of environmental and perceptual uncertainty using deterministic reinforcement learning dynamics with partial observability](https://mlsanity.com/paper/2109.07259/) (2021)
- [Intrinsic fluctuations of reinforcement learning promote cooperation](https://mlsanity.com/paper/2209.01013/) (2022)
- [Self-Clustering Hierarchical Multi-Agent Reinforcement Learning with Extensible Cooperation Graph](https://mlsanity.com/paper/2403.18056/) (2024)
- [How social reinforcement learning can lead to metastable polarisation and the voter model](https://mlsanity.com/paper/2406.07993/) (2024)
- [Artificial Utopia: Simulation and Intelligent Agents for a Democratised Future](https://mlsanity.com/paper/2503.07364/) (2025)

## Abstract (arXiv)

Reinforcement learning in multiagent systems has been studied in the fields of economic game theory, artificial intelligence and statistical physics by developing an analytical understanding of the learning dynamics (often in relation to the replicator dynamics of evolutionary game theory). However, the majority of these analytical studies focuses on repeated normal form games, which only have a single environmental state. Environmental dynamics, i.e., changes in the state of an environment affecting the agents' payoffs has received less attention, lacking a universal method to obtain deterministic equations from established multistate reinforcement learning algorithms. In this work we present a novel methodological extension, separating the interaction from the adaptation time scale, to derive the deterministic limit of a general class of reinforcement learning algorithms, called temporal difference learning. This form of learning is equipped to function in more realistic multistate environments by using the estimated value of future environmental states to adapt the agent's behavior. We demonstrate the potential of our method with the three well established learning algorithms Q learning, SARSA learning and Actor-Critic learning. Illustrations of their dynamics on two multiagent, multistate environments reveal a wide range of different dynamical regimes, such as convergence to fixed points, limit cycles, and even deterministic chaos.