Reinforcement Learning for Control Systems with Time Delays: A Comprehensive Survey
Not on the map. Why?
Abstract (arXiv)
In the last decade, Reinforcement Learning (RL) has achieved remarkable success in the control and decision-making of complex dynamical systems. However, most RL algorithms rely on the Markov Decision Process assumption, which is violated in practical cyber-physical systems affected by sensing delays, actuation latencies, and communication constraints. Such time delays introduce memory effects that can significantly degrade performance and compromise stability, particularly in networked and multi-agent environments. This paper presents a comprehensive survey of RL methods designed to address time delays in control systems. We first formalize the main classes of delays and analyze their impact on the Markov property. We then systematically categorize existing approaches into five major families: state augmentation and history-based representations, recurrent policies with learned memory, predictor-based and model-aware methods, robust and domain-randomized training strategies, and safe RL frameworks with explicit constraint handling. For each family, we discuss underlying principles, practical advantages, and inherent limitations. A comparative analysis highlights key trade-offs among these approaches and provides practical guidelines for selecting suitable methods under different delay characteristics and safety requirements. Finally, we identify open challenges and promising research directions, including stability certification, large-delay learning, multi-agent communication co-design, and standardized benchmarking. This survey aims to serve as a unified reference for researchers and practitioners developing reliable RL-based controllers in delay-affected cyber-physical systems.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 46 of its 76 references that are in the map. First posted on arXiv 30 Jan 2026.
Builds on
46 of 76 references.
References · 46
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
- 2015Continuous control with deep reinforcement learningcited by 3,279
- 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
- 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
- 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
- 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
- 2017Constrained Policy Optimizationcited by 928
- 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
- 2016Learning to Communicate with Deep Multi-Agent Reinforcement Learningcited by 561
- 2019Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmscited by 555
- 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 457
34 more
- 2019Control Barrier Functions: Theory and Applicationscited by 332
- 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 278
- 2018Safe Exploration in Continuous Action Spacescited by 237
- 2015A Critical Review of Recurrent Neural Networks for Sequence Learningcited by 223
- 2018Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experiencecited by 203
- 2017Robust Deep Reinforcement Learning with Adversarial Attackscited by 152
- 2021Online Robust Reinforcement Learning with Model Uncertaintycited by 95
- 2021Sample Complexity of Robust Reinforcement Learning with a Generative Modelcited by 82
- 2015Memory-based control with recurrent neural networkscited by 77
- 2019Learning for Safety-Critical Control with Control Barrier Functionscited by 66
- 2022A Survey on Model-based Reinforcement Learningcited by 65
- 2017OptLayer - Practical Constrained Optimization for Deep Reinforcement Learning in the Real Worldcited by 64
- 2020Robust Reinforcement Learning using Adversarial Populationscited by 64
- 2020Reinforcement Learning with Random Delayscited by 56
- 2021Safe Reinforcement Learning Using Robust Control Barrier Functionscited by 44
- 2019Real-Time Reinforcement Learningcited by 38
- 2020Delay-Aware Model-Based Reinforcement Learning for Continuous Controlcited by 33
- 2022Robust Reinforcement Learning as a Stackelberg Game via Adaptively-Regularized Adversarial Trainingcited by 26
- 2020Delay-Aware Multi-Agent Reinforcement Learning for Cooperative and Competitive Environmentscited by 21
- 2021Revisiting State Augmentation methods for Reinforcement Learning with Stochastic Delayscited by 20
- 2021Recurrent Off-policy Baselines for Memory-based Continuous Controlcited by 19
- 2020Blind Decision Making: Reinforcement Learning with Delayed Observationscited by 14
- 2019Robust Visual Domain Randomization for Reinforcement Learningcited by 13
- 2024Reinforcement Learning from Delayed Observations via World Modelscited by 13
- 2025A Survey of Safe Reinforcement Learning and Constrained MDPs: A Technical Survey on Single-Agent and Multi-Agent Safetycited by 12
- 2020Robust Domain Randomised Reinforcement Learning through Peer-to-Peer Distillationcited by 10
- 2022DACOM: Learning Delay-Aware Communication for Multi-Agent Reinforcement Learningcited by 9
- 2024Variational Delayed Policy Optimizationcited by 9
- 2022Generalization in Deep Reinforcement Learning for Robotic Navigation by Reward Shapingcited by 8
- 2024DEER: A Delay-Resilient Framework for Reinforcement Learning with Variable Delayscited by 6
- 2023Efficient Reinforcement Learning with Impaired Observability: Learning to Act with Delayed and Missing State Observationscited by 4
- 2023DiAReL: Reinforcement Learning with Disturbance Awareness for Robust Sim2Real Policy Transfer in Robot Controlcited by 4
- 2025Directly Forecasting Belief for Reinforcement Learning with Delayscited by 4
- 2025Model-Based Reinforcement Learning under Random Observation Delayscited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.