# Delay-Aware Model-Based Reinforcement Learning for Continuous Control

arXiv: https://arxiv.org/abs/2005.05440 · HTML: https://mlsanity.com/paper/2005.05440/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 May 2020.

## Built on by (most cited first, 30 of 34)

- [Context-Aware Safe Reinforcement Learning for Non-Stationary Environments](https://mlsanity.com/paper/2101.00531/) (2021)
- [Acting in Delayed Environments with Non-Stationary Markov Policies](https://mlsanity.com/paper/2101.11992/) (2021)
- [Robust Reinforcement Learning as a Stackelberg Game via Adaptively-Regularized Adversarial Training](https://mlsanity.com/paper/2202.09514/) (2022)
- [Revisiting State Augmentation methods for Reinforcement Learning with Stochastic Delays](https://mlsanity.com/paper/2108.07555/) (2021)
- [Asynchronous Multi-Agent Reinforcement Learning for Efficient Real-Time Multi-Robot Cooperative Exploration](https://mlsanity.com/paper/2301.03398/) (2023)
- [Delayed Reinforcement Learning by Imitation](https://mlsanity.com/paper/2205.05569/) (2022)
- [Real-Time Reasoning Agents in Evolving Environments](https://mlsanity.com/paper/2511.04898/) (2025)
- [Reinforcement Learning from Delayed Observations via World Models](https://mlsanity.com/paper/2403.12309/) (2024)
- [Neural Laplace Control for Continuous-time Delayed Systems](https://mlsanity.com/paper/2302.12604/) (2023)
- [Boosting Reinforcement Learning with Strongly Delayed Feedback Through Auxiliary Short Delays](https://mlsanity.com/paper/2402.03141/) (2024)
- [DACOM: Learning Delay-Aware Communication for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2212.01619/) (2022)
- [Variational Delayed Policy Optimization](https://mlsanity.com/paper/2405.14226/) (2024)
- [DiAReL: Reinforcement Learning with Disturbance Awareness for Robust Sim2Real Policy Transfer in Robot Control](https://mlsanity.com/paper/2306.09010/) (2023)
- [DEER: A Delay-Resilient Framework for Reinforcement Learning with Variable Delays](https://mlsanity.com/paper/2406.03102/) (2024)
- [Reducing the Cost of Cycle-Time Tuning for Real-World Policy Optimization](https://mlsanity.com/paper/2305.05760/) (2023)
- [Evaluating the progress of Deep Reinforcement Learning in the real world: aligning domain-agnostic and domain-specific research](https://mlsanity.com/paper/2107.03015/) (2021)
- [Temporally Layered Architecture for Adaptive, Distributed and Continuous Control](https://mlsanity.com/paper/2301.00723/) (2022)
- [Adaptive PD Control using Deep Reinforcement Learning for Local-Remote Teleoperation with Stochastic Time Delays](https://mlsanity.com/paper/2305.16979/) (2023)
- [Control in Stochastic Environment with Delays: A Model-based Reinforcement Learning Approach](https://mlsanity.com/paper/2402.00313/) (2024)
- [Belief-Based Offline Reinforcement Learning for Delay-Robust Policy Optimization](https://mlsanity.com/paper/2506.00131/) (2025)
- [Reinforcement Learning via Conservative Agent for Environments with Random Delays](https://mlsanity.com/paper/2507.18992/) (2025)
- [RT-HCP: Dealing with Inference Delays and Sample Efficiency to Learn Directly on Robotic Platforms](https://mlsanity.com/paper/2509.06714/) (2025)
- [Reinforcement Learning for Control Systems with Time Delays: A Comprehensive Survey](https://mlsanity.com/paper/2602.00399/) (2026)
- [Trajectory Planning for Teleoperated Space Manipulators Using Deep Reinforcement Learning](https://mlsanity.com/paper/2408.05460/) (2024)
- [Overcoming Slow Decision Frequencies in Continuous Control: Model-Based Sequence Reinforcement Learning for Model-Free Control](https://mlsanity.com/paper/2410.08979/) (2024)
- [Patrol Security Game: Defending Against Adversary with Freedom in Attack Timing, Location, and Duration](https://mlsanity.com/paper/2410.15600/) (2024)
- [Inverse Delayed Reinforcement Learning](https://mlsanity.com/paper/2412.02931/) (2024)
- [Rainbow Delay Compensation: A Multi-Agent Reinforcement Learning Framework for Mitigating Delayed Observation](https://mlsanity.com/paper/2505.03586/) (2025)
- [Intelligent Spectrum Management in Satellite Communications](https://mlsanity.com/paper/2509.00286/) (2025)
- [State-Action Inpainting Diffuser for Continuous Control with Delay](https://mlsanity.com/paper/2603.01553/) (2026)

## Abstract (arXiv)

Action delays degrade the performance of reinforcement learning in many real-world systems. This paper proposes a formal definition of delay-aware Markov Decision Process and proves it can be transformed into standard MDP with augmented states using the Markov reward process. We develop a delay-aware model-based reinforcement learning framework that can incorporate the multi-step delay into the learned system models without learning effort. Experiments with the Gym and MuJoCo platforms show that the proposed delay-aware model-based algorithm is more efficient in training and transferable between systems with various durations of delay compared with off-policy model-free reinforcement learning methods. Codes available at: https://github.com/baimingc/dambrl.