# An Evolutionary Computation Framework for Multi-Agent Q-Learning with Mean-Field Environmental Feedback

arXiv: https://arxiv.org/abs/2609.13253 · HTML: https://mlsanity.com/paper/2609.13253/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 55 references that are in the map.
- First posted on arXiv 5 Sep 2026.

## Builds on (most cited first, 10 of 10)

- [Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms](https://mlsanity.com/paper/1911.10635/) (2019)
- [A Survey and Critique of Multiagent Deep Reinforcement Learning](https://mlsanity.com/paper/1810.05587/) (2018)
- [Networks beyond pairwise interactions: structure and dynamics](https://mlsanity.com/paper/2006.01764/) (2020)
- [Statistical physics of human cooperation](https://mlsanity.com/paper/1705.07161/) (2017)
- [The Dynamics of Q-learning in Population Games: a Physics-Inspired Continuity Equation Model](https://mlsanity.com/paper/2203.01500/) (2022)
- [Punishment and inspection for governing the commons in a feedback-evolving game](https://mlsanity.com/paper/1807.05484/) (2018)
- [Limit Cycles Analysis and Control of Evolutionary Game Dynamics with Environmental Feedback](https://mlsanity.com/paper/2205.10734/) (2022)
- [Interdependent network reciprocity in evolutionary games](https://mlsanity.com/paper/1308.1947/) (2013)
- [Dynamic Evolution of Complex Networks: A Reinforcement Learning Approach Applying Evolutionary Games to Community Structure](https://mlsanity.com/paper/2506.17925/) (2025)
- [Evolutionary Dynamics with Self-Interaction Learning in Networked Systems](https://mlsanity.com/paper/2507.00422/) (2025)

## Abstract (arXiv)

Multi-agent reinforcement learning in networked populations is governed by the interaction between individual adaptation, local encounters, and changing environmental conditions. To study this interaction, we formulate a coupled learning--environment model in which agents update stateless Q-values on a fixed graph, while their population-average behavior drives an environmental variable that dynamically modifies the payoff matrix. Under a first-order mean-field closure, we derive a deterministic transport equation for the population distribution of Q-values and couple it with a projected discrete update for the environmental state. The resulting model is evaluated against finite-network Monte Carlo simulations on random regular, ErdHos--R'enyi, Barab'asi--Albert, and random geometric graphs. Across the tested parameter ranges, the mean-field system reproduces the main macroscopic cooperation and environmental trajectories, and the trajectory-level root-mean-square error generally decreases with population size and average degree. The analysis further shows that environmental feedback reshapes the learned action-value ordering, while reinforcing feedback can produce pronounced dependence on the initial learning bias and resource level. The environmental timescale also plays an important role: a rapid response can drive the resource state to a boundary before learning adapts, whereas a slower response preserves the interaction between behavioral learning and environmental recovery. These results provide a population-level description of coupled reinforcement learning and environmental dynamics and characterize the performance of the mean-field approximation within the tested network and parameter ranges.