# Learning Individual Policies in Large Multi-agent Systems through Local Variance Minimization

arXiv: https://arxiv.org/abs/2212.13379 · HTML: https://mlsanity.com/paper/2212.13379/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 53 references that are in the map.
- First posted on arXiv 27 Dec 2022.

## Builds on (most cited first, 16 of 16)

- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1803.11485/) (2018)
- [Value-Decomposition Networks For Cooperative Multi-Agent Learning](https://mlsanity.com/paper/1706.05296/) (2017)
- [Mean Field Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1802.05438/) (2018)
- [Deep Reinforcement Learning from Self-Play in Imperfect-Information Games](https://mlsanity.com/paper/1603.01121/) (2016)
- [Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1702.08887/) (2017)
- [Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharing](https://mlsanity.com/paper/2102.07475/) (2021)
- [Approximately Solving Mean Field Games via Entropy-Regularized Deep Reinforcement Learning](https://mlsanity.com/paper/2102.01585/) (2021)
- [Deep Reinforcement Learning for Swarm Systems](https://mlsanity.com/paper/1807.06613/) (2018)
- [Neural Approximate Dynamic Programming for On-Demand Ride-Pooling](https://mlsanity.com/paper/1911.08842/) (2019)
- [Multi Type Mean Field Reinforcement Learning](https://mlsanity.com/paper/2002.02513/) (2020)
- [Decentralized Mean Field Games](https://mlsanity.com/paper/2112.09099/) (2021)
- [Variance Penalized On-Policy and Off-Policy Actor-Critic](https://mlsanity.com/paper/2102.01985/) (2021)
- [Stochastic Potential Games](https://mlsanity.com/paper/2005.13527/) (2020)
- [Entropy based Independent Learning in Anonymous Multi-Agent Settings](https://mlsanity.com/paper/1803.09928/) (2018)
- [Learning Nash Equilibria in Congestion Games](https://mlsanity.com/paper/1408.0017/) (2014)

## Abstract (arXiv)

In multi-agent systems with large number of agents, typically the contribution of each agent to the value of other agents is minimal (e.g., aggregation systems such as Uber, Deliveroo). In this paper, we consider such multi-agent systems where each agent is self-interested and takes a sequence of decisions and represent them as a Stochastic Non-atomic Congestion Game (SNCG). We derive key properties for equilibrium solutions in SNCG model with non-atomic and also nearly non-atomic agents. With those key equilibrium properties, we provide a novel Multi-Agent Reinforcement Learning (MARL) mechanism that minimizes variance across values of agents in the same state. To demonstrate the utility of this new mechanism, we provide detailed results on a real-world taxi dataset and also a generic simulator for aggregation systems. We show that our approach reduces the variance in revenues earned by taxi drivers, while still providing higher joint revenues than leading approaches.