# Implementing Reinforcement Learning Datacenter Congestion Control in NVIDIA NICs

arXiv: https://arxiv.org/abs/2207.02295 · HTML: https://mlsanity.com/paper/2207.02295/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 64 references that are in the map.
- First posted on arXiv 5 Jul 2022.

## Builds on (most cited first, 15 of 15)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Policy Distillation](https://mlsanity.com/paper/1511.06295/) (2015)
- [Agent57: Outperforming the Atari Human Benchmark](https://mlsanity.com/paper/2003.13350/) (2020)
- [A Survey on the Explainability of Supervised Machine Learning](https://mlsanity.com/paper/2011.07876/) (2020)
- [Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation](https://mlsanity.com/paper/2004.09602/) (2020)
- [A Deep Hierarchical Approach to Lifelong Learning in Minecraft](https://mlsanity.com/paper/1604.07255/) (2016)
- [Improving the Interpretability of Deep Neural Networks with Knowledge Distillation](https://mlsanity.com/paper/1812.10924/) (2018)
- [Model Distillation for Revenue Optimization: Interpretable Personalized Pricing](https://mlsanity.com/paper/2007.01903/) (2020)
- [Verifying Robustness of Gradient Boosted Models](https://mlsanity.com/paper/1906.10991/) (2019)
- [Reinforcement Learning for Datacenter Congestion Control](https://mlsanity.com/paper/2102.09337/) (2021)
- [When Machine Learning Meets Congestion Control: A Survey and Comparison](https://mlsanity.com/paper/2010.11397/) (2020)
- [Interpreting Deep Learning-Based Networking Systems](https://mlsanity.com/paper/1910.03835/) (2019)
- [Robust Congestion Control for Demand-Based Optimization in Precoded Multi-Beam High Throughput Satellite Communications](https://mlsanity.com/paper/2109.02327/) (2021)

## Built on by (most cited first, 2 of 2)

- [X-RLflow: Graph Reinforcement Learning for Neural Network Subgraphs Transformation](https://mlsanity.com/paper/2304.14698/) (2023)
- [RayNet: A Simulation Platform for Developing Reinforcement Learning-Driven Network Protocols](https://mlsanity.com/paper/2302.04519/) (2023)

## Abstract (arXiv)

As communication protocols evolve, datacenter network utilization increases. As a result, congestion is more frequent, causing higher latency and packet loss. Combined with the increasing complexity of workloads, manual design of congestion control (CC) algorithms becomes extremely difficult. This calls for the development of AI approaches to replace the human effort. Unfortunately, it is currently not possible to deploy AI models on network devices due to their limited computational capabilities. Here, we offer a solution to this problem by building a computationally-light solution based on a recent reinforcement learning CC algorithm [arXiv:2207.02295]. We reduce the inference time of RL-CC by x500 by distilling its complex neural network into decision trees. This transformation enables real-time inference within the μ-sec decision-time requirement, with a negligible effect on quality. We deploy the transformed policy on NVIDIA NICs in a live cluster. Compared to popular CC algorithms used in production, RL-CC is the only method that performs well on all benchmarks tested over a large range of number of flows. It balances multiple metrics simultaneously: bandwidth, latency, and packet drops. These results suggest that data-driven methods for CC are feasible, challenging the prior belief that handcrafted heuristics are necessary to achieve optimal performance.