# Homogenization of Multi-agent Learning Dynamics in Finite-state Markov Games

arXiv: https://arxiv.org/abs/2506.21079 · HTML: https://mlsanity.com/paper/2506.21079/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 2 of its 12 references that are in the map.
- First posted on arXiv 26 Jun 2025.

## Builds on (most cited first, 2 of 2)

- [Artificial Intelligence and Spontaneous Collusion](https://mlsanity.com/paper/2202.05946/) (2022)
- [Computing the Bias of Constant-step Stochastic Approximation with Markovian Noise](https://mlsanity.com/paper/2405.14285/) (2024)

## Abstract (arXiv)

This paper introduces a new approach for approximating the learning dynamics of multiple reinforcement learning (RL) agents interacting in a finite-state Markov game. The idea is to rescale the learning process by simultaneously reducing the learning rate and increasing the update frequency, effectively treating the agent's parameters as a slow-evolving variable influenced by the fast-mixing game state. Under mild assumptions-ergodicity of the state process and continuity of the updates-we prove the convergence of this rescaled process to an ordinary differential equation (ODE). This ODE provides a tractable, deterministic approximation of the agent's learning dynamics. An implementation of the framework is available at,: https://github.com/yannKerzreho/MarkovGameApproximation