# Delayed Reinforcement Learning by Imitation

arXiv: https://arxiv.org/abs/2205.05569 · HTML: https://mlsanity.com/paper/2205.05569/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 44 references that are in the map.
- First posted on arXiv 11 May 2022.

## Builds on (most cited first, 18 of 18)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Masked Autoregressive Flow for Density Estimation](https://mlsanity.com/paper/1705.07057/) (2017)
- [An Algorithmic Perspective on Imitation Learning](https://mlsanity.com/paper/1811.06711/) (2018)
- [Challenges of Real-World Reinforcement Learning](https://mlsanity.com/paper/1904.12901/) (2019)
- [DART: Noise Injection for Robust Imitation Learning](https://mlsanity.com/paper/1703.09327/) (2017)
- [Error Bounds of Imitating Policies and Environments](https://mlsanity.com/paper/2010.11876/) (2020)
- [Reinforcement Learning with Random Delays](https://mlsanity.com/paper/2010.02966/) (2020)
- [Regularizing Action Policies for Smooth Control with Reinforcement Learning](https://mlsanity.com/paper/2012.06644/) (2020)
- [Delay-Aware Model-Based Reinforcement Learning for Continuous Control](https://mlsanity.com/paper/2005.05440/) (2020)
- [Setting up a Reinforcement Learning Task with a Real-World Robot](https://mlsanity.com/paper/1803.07067/) (2018)
- [Acting in Delayed Environments with Non-Stationary Markov Policies](https://mlsanity.com/paper/2101.11992/) (2021)
- [Control Frequency Adaptation via Action Persistence in Batch Reinforcement Learning](https://mlsanity.com/paper/2002.06836/) (2020)
- [At Human Speed: Deep Reinforcement Learning with Action Delay](https://mlsanity.com/paper/1810.07286/) (2018)
- [Blind Decision Making: Reinforcement Learning with Delayed Observations](https://mlsanity.com/paper/2011.07715/) (2020)

## Built on by (most cited first, 16 of 16)

- [Reinforcement Learning from Delayed Observations via World Models](https://mlsanity.com/paper/2403.12309/) (2024)
- [Neural Laplace Control for Continuous-time Delayed Systems](https://mlsanity.com/paper/2302.12604/) (2023)
- [Boosting Reinforcement Learning with Strongly Delayed Feedback Through Auxiliary Short Delays](https://mlsanity.com/paper/2402.03141/) (2024)
- [Variational Delayed Policy Optimization](https://mlsanity.com/paper/2405.14226/) (2024)
- [No-Regret Reinforcement Learning in Smooth MDPs](https://mlsanity.com/paper/2402.03792/) (2024)
- [Projection by Convolution: Optimal Sample Complexity for Reinforcement Learning in Continuous-Space MDPs](https://mlsanity.com/paper/2405.06363/) (2024)
- [Local Linearity: the Key for No-regret Reinforcement Learning in Continuous MDPs](https://mlsanity.com/paper/2410.24071/) (2024)
- [Model-Based Reinforcement Learning under Random Observation Delays](https://mlsanity.com/paper/2509.20869/) (2025)
- [Belief-Based Offline Reinforcement Learning for Delay-Robust Policy Optimization](https://mlsanity.com/paper/2506.00131/) (2025)
- [Reinforcement Learning via Conservative Agent for Environments with Random Delays](https://mlsanity.com/paper/2507.18992/) (2025)
- [Reinforcement Learning with Action-Triggered Observations](https://mlsanity.com/paper/2510.02149/) (2025)
- [Trajectory Planning for Teleoperated Space Manipulators Using Deep Reinforcement Learning](https://mlsanity.com/paper/2408.05460/) (2024)
- [Inverse Delayed Reinforcement Learning](https://mlsanity.com/paper/2412.02931/) (2024)
- [Rainbow Delay Compensation: A Multi-Agent Reinforcement Learning Framework for Mitigating Delayed Observation](https://mlsanity.com/paper/2505.03586/) (2025)
- [State-Action Inpainting Diffuser for Continuous Control with Delay](https://mlsanity.com/paper/2603.01553/) (2026)
- [Delay-Empowered Causal Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2605.12261/) (2026)

## Abstract (arXiv)

When the agent's observations or interactions are delayed, classic reinforcement learning tools usually fail. In this paper, we propose a simple yet new and efficient solution to this problem. We assume that, in the undelayed environment, an efficient policy is known or can be easily learned, but the task may suffer from delays in practice and we thus want to take them into account. We present a novel algorithm, Delayed Imitation with Dataset Aggregation (DIDA), which builds upon imitation learning methods to learn how to act in a delayed environment from undelayed demonstrations. We provide a theoretical analysis of the approach that will guide the practical design of DIDA. These results are also of general interest in the delayed reinforcement learning literature by providing bounds on the performance between delayed and undelayed tasks, under smoothness conditions. We show empirically that DIDA obtains high performances with a remarkable sample efficiency on a variety of tasks, including robotic locomotion, classic control, and trading.