Implementing Reinforcement Learning Datacenter Congestion Control in NVIDIA NICs
Not on the map. Why?
Abstract (arXiv)
As communication protocols evolve, datacenter network utilization increases. As a result, congestion is more frequent, causing higher latency and packet loss. Combined with the increasing complexity of workloads, manual design of congestion control (CC) algorithms becomes extremely difficult. This calls for the development of AI approaches to replace the human effort. Unfortunately, it is currently not possible to deploy AI models on network devices due to their limited computational capabilities. Here, we offer a solution to this problem by building a computationally-light solution based on a recent reinforcement learning CC algorithm [arXiv:2207.02295]. We reduce the inference time of RL-CC by x500 by distilling its complex neural network into decision trees. This transformation enables real-time inference within the μ-sec decision-time requirement, with a negligible effect on quality. We deploy the transformed policy on NVIDIA NICs in a live cluster. Compared to popular CC algorithms used in production, RL-CC is the only method that performs well on all benchmarks tested over a large range of number of flows. It balances multiple metrics simultaneously: bandwidth, latency, and packet drops. These results suggest that data-driven methods for CC are feasible, challenging the prior belief that handcrafted heuristics are necessary to achieve optimal performance.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 15 of its 64 references that are in the map. First posted on arXiv 5 Jul 2022.
Builds on
15 of 64 references.
References · 15
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2015Continuous control with deep reinforcement learningcited by 3,279
- 2015Policy Distillationcited by 414
- 2020Agent57: Outperforming the Atari Human Benchmarkcited by 300
- 2020A Survey on the Explainability of Supervised Machine Learningcited by 225
- 2020Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluationcited by 183
- 2016A Deep Hierarchical Approach to Lifelong Learning in Minecraftcited by 90
- 2018Improving the Interpretability of Deep Neural Networks with Knowledge Distillationcited by 37
- 2020Model Distillation for Revenue Optimization: Interpretable Personalized Pricingcited by 24
- 2019Verifying Robustness of Gradient Boosted Modelscited by 13
- 2021Reinforcement Learning for Datacenter Congestion Controlcited by 11
Built on by
Citing papers per year: 2023: 2.
2 papers.