# Learning Mean-Field Control for Delayed Information Load Balancing in Large Queuing Systems

arXiv: https://arxiv.org/abs/2208.04777 · HTML: https://mlsanity.com/paper/2208.04777/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 45 references that are in the map.
- First posted on arXiv 9 Aug 2022.

## Builds on (most cited first, 13 of 13)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms](https://mlsanity.com/paper/1911.10635/) (2019)
- [Applications of Deep Reinforcement Learning in Communications and Networking: A Survey](https://mlsanity.com/paper/1810.07862/) (2018)
- [Approximately Solving Mean Field Games via Entropy-Regularized Deep Reinforcement Learning](https://mlsanity.com/paper/2102.01585/) (2021)
- [Model-Free Mean-Field Reinforcement Learning: Mean-Field MDP and Mean-Field Q-Learning](https://mlsanity.com/paper/1910.12802/) (2019)
- [Mean-Field Controls with Q-learning for Cooperative MARL: Convergence and Complexity Analysis](https://mlsanity.com/paper/2002.04131/) (2020)
- [On the Approximation of Cooperative Heterogeneous Multi-Agent Reinforcement Learning (MARL) using Mean Field Control (MFC)](https://mlsanity.com/paper/2109.04024/) (2021)
- [Markov-Nash Equilibria in Mean-Field Games with Discounted Cost](https://mlsanity.com/paper/1612.07878/) (2016)
- [Discrete-Time Mean Field Control with Environment States](https://mlsanity.com/paper/2104.14900/) (2021)
- [McKean-Vlasov optimal control: the dynamic programming principle](https://mlsanity.com/paper/1907.08860/) (2019)
- [Team Optimal Control of Coupled Subsystems with Mean-Field Sharing](https://mlsanity.com/paper/2012.01418/) (2020)
- [An Overview for Markov Decision Processes in Queues and Networks](https://mlsanity.com/paper/1907.10243/) (2019)
- [Asymptotically Optimal Load Balancing in Large-scale Heterogeneous Systems with Multiple Dispatchers](https://mlsanity.com/paper/2002.08908/) (2020)

## Built on by (most cited first, 2 of 2)

- [A Survey on Large-Population Systems and Scalable Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2209.03859/) (2022)
- [Sparse Mean Field Load Balancing in Large Localized Queueing Systems](https://mlsanity.com/paper/2312.12973/) (2023)

## Abstract (arXiv)

Recent years have seen a great increase in the capacity and parallel processing power of data centers and cloud services. To fully utilize the said distributed systems, optimal load balancing for parallel queuing architectures must be realized. Existing state-of-the-art solutions fail to consider the effect of communication delays on the behaviour of very large systems with many clients. In this work, we consider a multi-agent load balancing system, with delayed information, consisting of many clients (load balancers) and many parallel queues. In order to obtain a tractable solution, we model this system as a mean-field control problem with enlarged state-action space in discrete time through exact discretization. Subsequently, we apply policy gradient reinforcement learning algorithms to find an optimal load balancing solution. Here, the discrete-time system model incorporates a synchronization delay under which the queue state information is synchronously broadcasted and updated at all clients. We then provide theoretical performance guarantees for our methodology in large systems. Finally, using experiments, we prove that our approach is not only scalable but also shows good performance when compared to the state-of-the-art power-of-d variant of the Join-the-Shortest-Queue (JSQ) and other policies in the presence of synchronization delays.