Demystifying Reinforcement Learning in Time-Varying Systems
Not on the map. Why?
Abstract (arXiv)
Recent research has turned to Reinforcement Learning (RL) to solve challenging decision problems, as an alternative to hand-tuned heuristics. RL can learn good policies without the need for modeling the environment's dynamics. Despite this promise, RL remains an impractical solution for many real-world systems problems. A particularly challenging case occurs when the environment changes over time, i.e. it exhibits non-stationarity. In this work, we characterize the challenges introduced by non-stationarity, shed light on the range of approaches to them and develop a robust framework for addressing them to train RL agents in live systems. Such agents must explore and learn new environments, without hurting the system's performance, and remember them over time. To this end, our framework (i) identifies different environments encountered by the live system, (ii) triggers exploration when necessary, (iii) takes precautions to retain knowledge from prior environments, and (iv) employs safeguards to protect the system's performance when the RL agent makes mistakes. We apply our framework to two systems problems, straggler mitigation and adaptive video streaming, and evaluate it against a variety of alternative approaches using real-world and synthetic data. We show that all components of the framework are necessary to cope with non-stationarity and provide guidance on alternative design choices for each component.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 40 of its 70 references that are in the map. First posted on arXiv 14 Jan 2022.
Builds on
40 of 70 references.
References · 40
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
- 2015Continuous control with deep reinforcement learningcited by 3,279
- 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
- 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
- 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
- 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
- 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
- 2016Progressive Neural Networkscited by 1,332
- 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
28 more
- 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
- 2018Experience Replay for Continual Learningcited by 1,074
- 2018Progress & Compress: A scalable framework for continual learningcited by 516
- 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
- 2018Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learningcited by 270
- 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 268
- 2018Selective Experience Replay for Lifelong Learningcited by 239
- 2018Safe Exploration in Continuous Action Spacescited by 237
- 2018Understanding the impact of entropy on policy optimizationcited by 171
- 2018Multi-task Deep Reinforcement Learning with PopArtcited by 138
- 2018Learning Scheduling Algorithms for Data Processing Clusterscited by 133
- 2017Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environmentscited by 106
- 2016Sim-to-Real Robot Learning from Pixels with Progressive Netscited by 89
- 2016Learning values across many orders of magnitudecited by 80
- 2018Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RLcited by 80
- 2019Neo: A Learned Query Optimizercited by 80
- 2016Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Criticcited by 77
- 2016Transfer from Simulation to Real World through Learning Deep Inverse Dynamics Modelcited by 53
- 2018Pseudo-Rehearsal: Achieving Deep Reinforcement Learning without Catastrophic Forgettingcited by 46
- 2018Learning to Optimize Join Queries With Deep Reinforcement Learningcited by 32
- 2018Continual Reinforcement Learning with Complex Synapsescited by 30
- 2020SAMBA: Safe Model-Based & Active Reinforcement Learningcited by 28
- 2016BOLA: Near-Optimal Bitrate Adaptation for Online Videoscited by 26
- 2020Towards Safe Policy Improvement for Non-Stationary MDPscited by 26
- 2021Minimum-Delay Adaptation in Non-Stationary Reinforcement Learning via Online High-Confidence Change-Point Detectioncited by 18
- 2017Optimal and Scalable Caching for 5G Using Reinforcement Learning of Space-time Popularitiescited by 15
- 2020Knowledge-Assisted Deep Reinforcement Learning in 5G Scheduler Design: From Theoretical Framework to Implementationcited by 8
- 2018Internet Congestion Control via Deep Reinforcement Learningcited by 7
Built on by
Citing papers per year: 2023: 1; 2024: 1.
2 papers.