# Plume: A Framework for High Performance Deep RL Network Controllers via Prioritized Trace Sampling

arXiv: https://arxiv.org/abs/2302.12403 · HTML: https://mlsanity.com/paper/2302.12403/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 61 references that are in the map.
- First posted on arXiv 24 Feb 2023.

## Builds on (most cited first, 21 of 21)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Deep Reinforcement Learning at the Edge of the Statistical Precipice](https://mlsanity.com/paper/2108.13264/) (2021)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [Distributed Prioritized Experience Replay](https://mlsanity.com/paper/1803.00933/) (2018)
- [Sample Efficient Actor-Critic with Experience Replay](https://mlsanity.com/paper/1611.01224/) (2016)
- [Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1702.08887/) (2017)
- [Identifying and Compensating for Feature Deviation in Imbalanced Deep Learning](https://mlsanity.com/paper/2001.01385/) (2020)
- [Language Understanding for Text-based Games Using Deep Reinforcement Learning](https://mlsanity.com/paper/1506.08941/) (2015)
- [RLlib: Abstractions for Distributed Reinforcement Learning](https://mlsanity.com/paper/1712.09381/) (2017)
- [Smooth Exploration for Robotic Reinforcement Learning](https://mlsanity.com/paper/2005.05719/) (2020)
- [Observe and Look Further: Achieving Consistent Performance on Atari](https://mlsanity.com/paper/1805.11593/) (2018)
- [Variance Reduction for Reinforcement Learning in Input-Driven Environments](https://mlsanity.com/paper/1807.02264/) (2018)
- [Real-world Video Adaptation with Reinforcement Learning](https://mlsanity.com/paper/2008.12858/) (2020)
- [Let's Play Again: Variability of Deep Reinforcement Learning Agents in Atari Environments](https://mlsanity.com/paper/1904.06312/) (2019)
- [Learning in situ: a randomized experiment in video streaming](https://mlsanity.com/paper/1906.01113/) (2019)

## Abstract (arXiv)

Deep Reinforcement Learning (DRL) has shown promise in various networking environments. However, these environments present several fundamental challenges for standard DRL techniques. They are difficult to explore and exhibit high levels of noise and uncertainty. Although these challenges complicate the training process, we find that in practice we can substantially mitigate their effects and even achieve state-of-the-art real-world performance by addressing a factor that has been previously overlooked: the skewed input trace distribution in DRL training datasets. We introduce a generalized framework, Plume, to automatically identify and balance the skew using a three-stage process. First, we identify the critical features that determine the behavior of the traces. Second, we classify the traces into clusters. Finally, we prioritize the salient clusters to improve the overall performance of the controller. Plume seamlessly works across DRL algorithms, without requiring any changes to the DRL workflow. We evaluated Plume on three networking environments, including Adaptive Bitrate Streaming, Congestion Control, and Load Balancing. Plume offers superior performance in both simulation and real-world settings, across different controllers and DRL algorithms. For example, our novel ABR controller, Gelato trained with Plume consistently outperforms prior state-of-the-art controllers on the live streaming platform Puffer for over a year. It is the first controller on the platform to deliver statistically significant improvements in both video quality and stalling, decreasing stalls by as much as 75%.