# Learning Distributed and Fair Policies for Network Load Balancing as Markov Potential Game

arXiv: https://arxiv.org/abs/2206.01451 · HTML: https://mlsanity.com/paper/2206.01451/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 72 references that are in the map.
- First posted on arXiv 3 Jun 2022.

## Builds on (most cited first, 24 of 24)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Counterfactual Multi-Agent Policy Gradients](https://mlsanity.com/paper/1705.08926/) (2017)
- [QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1803.11485/) (2018)
- [Value-Decomposition Networks For Cooperative Multi-Agent Learning](https://mlsanity.com/paper/1706.05296/) (2017)
- [Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?](https://mlsanity.com/paper/2011.09533/) (2020)
- [Learning Scheduling Algorithms for Data Processing Clusters](https://mlsanity.com/paper/1810.01963/) (2018)
- [Game-Theoretic Multiagent Reinforcement Learning](https://mlsanity.com/paper/2011.00583/) (2020)
- [Global Convergence of Multi-Agent Policy Gradient in Markov Potential Games](https://mlsanity.com/paper/2106.01969/) (2021)
- [Learning Individually Inferred Communication for Multi-Agent Cooperation](https://mlsanity.com/paper/2006.06455/) (2020)
- [Efficient Communication in Multi-Agent Reinforcement Learning via Variance Based Control](https://mlsanity.com/paper/1909.02682/) (2019)
- [Learning Fairness in Multi-Agent Systems](https://mlsanity.com/paper/1910.14472/) (2019)
- [Learning Fair Policies in Decentralized Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2012.09421/) (2020)
- [Independent Natural Policy Gradient Always Converges in Markov Potential Games](https://mlsanity.com/paper/2110.10614/) (2021)
- [Learning in Nonzero-Sum Stochastic Games with Potentials](https://mlsanity.com/paper/2103.09284/) (2021)
- [Microservices: yesterday, today, and tomorrow](https://mlsanity.com/paper/1606.04036/) (2016)
- [Learning Parametric Closed-Loop Policies for Markov Potential Games](https://mlsanity.com/paper/1802.00899/) (2018)
- [Multi-Agent Trust Region Policy Optimization](https://mlsanity.com/paper/2010.07916/) (2020)
- [MVFST-RL: An Asynchronous RL Framework for Congestion Control with Delayed Actions](https://mlsanity.com/paper/1910.04054/) (2019)
- [In-network Congestion-aware Load Balancing at Transport Layer](https://mlsanity.com/paper/1811.09731/) (2018)
- [Distributed Dispatching in the Parallel Server Model](https://mlsanity.com/paper/2008.00793/) (2020)
- [Spotlight: Scalable Transport Layer Load Balancing for Data Center Networks](https://mlsanity.com/paper/1806.08455/) (2018)
- [LB Scalability: Achieving the Right Balance Between Being Stateful and Stateless](https://mlsanity.com/paper/2010.13385/) (2020)

## Built on by (most cited first, 2 of 2)

- [FightLadder: A Benchmark for Competitive Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2406.02081/) (2024)
- [Independent Learning in Constrained Markov Potential Games](https://mlsanity.com/paper/2402.17885/) (2024)

## Abstract (arXiv)

This paper investigates the network load balancing problem in data centers (DCs) where multiple load balancers (LBs) are deployed, using the multi-agent reinforcement learning (MARL) framework. The challenges of this problem consist of the heterogeneous processing architecture and dynamic environments, as well as limited and partial observability of each LB agent in distributed networking systems, which can largely degrade the performance of in-production load balancing algorithms in real-world setups. Centralised-training-decentralised-execution (CTDE) RL scheme has been proposed to improve MARL performance, yet it incurs -- especially in distributed networking systems, which prefer distributed and plug-and-play design scheme -- additional communication and management overhead among agents. We formulate the multi-agent load balancing problem as a Markov potential game, with a carefully and properly designed workload distribution fairness as the potential function. A fully distributed MARL algorithm is proposed to approximate the Nash equilibrium of the game. Experimental evaluations involve both an event-driven simulator and real-world system, where the proposed MARL load balancing algorithm shows close-to-optimal performance in simulations, and superior results over in-production LBs in the real-world system.