# Stability of Multi-Agent Learning in Competitive Networks: Delaying the Onset of Chaos

arXiv: https://arxiv.org/abs/2312.11943 · HTML: https://mlsanity.com/paper/2312.11943/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 51 references that are in the map.
- First posted on arXiv 19 Dec 2023.

## Builds on (most cited first, 10 of 10)

- [Cycles in adversarial regularized learning](https://mlsanity.com/paper/1709.02738/) (2017)
- [Triple Generative Adversarial Nets](https://mlsanity.com/paper/1703.02291/) (2017)
- [Exploration-Exploitation in Multi-Agent Competition: Convergence with Bounded Rationality](https://mlsanity.com/paper/2106.12928/) (2021)
- [Fast and Furious Learning in Zero-Sum Games: Vanishing Regret with Non-Vanishing Step Sizes](https://mlsanity.com/paper/1905.04532/) (2019)
- [Transition to chaos in random neuronal networks](https://mlsanity.com/paper/1508.06486/) (2015)
- [Asymptotic Convergence and Performance of Multi-Agent Q-Learning Dynamics](https://mlsanity.com/paper/2301.09619/) (2023)
- [Learning in nonatomic games, Part I: Finite action spaces and population games](https://mlsanity.com/paper/2107.01595/) (2021)
- [Leader-Follower Network Aggregative Game with Stochastic Agents' Communication and Activeness](https://mlsanity.com/paper/1908.02458/) (2019)
- [Exponential Convergence of Gradient Methods in Concave Network Zero-sum Games](https://mlsanity.com/paper/2007.05477/) (2020)
- [Generalized Hamiltonian Dynamics and Chaos in Evolutionary Games on Networks](https://mlsanity.com/paper/2006.00404/) (2020)

## Abstract (arXiv)

The behaviour of multi-agent learning in competitive network games is often studied within the context of zero-sum games, in which convergence guarantees may be obtained. However, outside of this class the behaviour of learning is known to display complex behaviours and convergence cannot be always guaranteed. Nonetheless, in order to develop a complete picture of the behaviour of multi-agent learning in competitive settings, the zero-sum assumption must be lifted. Motivated by this we study the Q-Learning dynamics, a popular model of exploration and exploitation in multi-agent learning, in competitive network games. We determine how the degree of competition, exploration rate and network connectivity impact the convergence of Q-Learning. To study generic competitive games, we parameterise network games in terms of correlations between agent payoffs and study the average behaviour of the Q-Learning dynamics across all games drawn from a choice of this parameter. This statistical approach establishes choices of parameters for which Q-Learning dynamics converge to a stable fixed point. Differently to previous works, we find that the stability of Q-Learning is explicitly dependent only on the network connectivity rather than the total number of agents. Our experiments validate these findings and show that, under certain network structures, the total number of agents can be increased without increasing the likelihood of unstable or chaotic behaviours.