mlsanityData to 7 Oct 2026

Delayed Reinforcement Learning by Imitation

arXiv 2205.05569 ↗First posted 11 May 2022cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

When the agent's observations or interactions are delayed, classic reinforcement learning tools usually fail. In this paper, we propose a simple yet new and efficient solution to this problem. We assume that, in the undelayed environment, an efficient policy is known or can be easily learned, but the task may suffer from delays in practice and we thus want to take them into account. We present a novel algorithm, Delayed Imitation with Dataset Aggregation (DIDA), which builds upon imitation learning methods to learn how to act in a delayed environment from undelayed demonstrations. We provide a theoretical analysis of the approach that will guide the practical design of DIDA. These results are also of general interest in the delayed reinforcement learning literature by providing bounds on the performance between delayed and undelayed tasks, under smoothness conditions. We show empirically that DIDA obtains high performances with a remarkable sample efficiency on a variety of tasks, including robotic locomotion, classic control, and trading.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 18 of its 44 references that are in the map. First posted on arXiv 11 May 2022.

Builds on

18 of 44 references.

References · 18

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  6. 2017Masked Autoregressive Flow for Density Estimationcited by 680
  7. 2018An Algorithmic Perspective on Imitation Learningcited by 455
  8. 2019Challenges of Real-World Reinforcement Learningcited by 282
  9. 2017DART: Noise Injection for Robust Imitation Learningcited by 187
  10. 2020Error Bounds of Imitating Policies and Environmentscited by 90
  11. 2020Reinforcement Learning with Random Delayscited by 56
  12. 2020Regularizing Action Policies for Smooth Control with Reinforcement Learningcited by 47
6 more
  1. 2020Delay-Aware Model-Based Reinforcement Learning for Continuous Controlcited by 33
  2. 2018Setting up a Reinforcement Learning Task with a Real-World Robotcited by 32
  3. 2021Acting in Delayed Environments with Non-Stationary Markov Policiescited by 28
  4. 2020Control Frequency Adaptation via Action Persistence in Batch Reinforcement Learningcited by 27
  5. 2018At Human Speed: Deep Reinforcement Learning with Action Delaycited by 24
  6. 2020Blind Decision Making: Reinforcement Learning with Delayed Observationscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 7; 2025: 5; 2026: 2.

15 papers.

Sort:

By area:

  1. 2024Reinforcement Learning from Delayed Observations via World Modelscited by 13
  2. 2023Neural Laplace Control for Continuous-time Delayed Systemscited by 10
  3. 2024Boosting Reinforcement Learning with Strongly Delayed Feedback Through Auxiliary Short Delayscited by 10
  4. 2024Variational Delayed Policy Optimizationcited by 9
  5. 2024No-Regret Reinforcement Learning in Smooth MDPscited by 6
  6. 2024Projection by Convolution: Optimal Sample Complexity for Reinforcement Learning in Continuous-Space MDPscited by 5
  7. 2024Local Linearity: the Key for No-regret Reinforcement Learning in Continuous MDPscited by 4
  8. 2025Model-Based Reinforcement Learning under Random Observation Delayscited by 2
  9. 2025Belief-Based Offline Reinforcement Learning for Delay-Robust Policy Optimizationcited by 1
  10. 2025Reinforcement Learning via Conservative Agent for Environments with Random Delayscited by 1
  11. 2025Reinforcement Learning with Action-Triggered Observationscited by 1
  12. 2024Inverse Delayed Reinforcement Learningcited by 0
3 more
  1. 2025Rainbow Delay Compensation: A Multi-Agent Reinforcement Learning Framework for Mitigating Delayed Observationcited by 0
  2. 2026State-Action Inpainting Diffuser for Continuous Control with Delaycited by 0
  3. 2026Delay-Empowered Causal Hierarchical Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Delay-Aware Diffusion Policy: Bridging the Observation-Execution Gap in Dynamic Tasks
  2. 2025The Pitfalls of Imitation Learning when Actions are Continuous
  3. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning
  4. 2020Episodic Self-Imitation Learning with Hindsight
  5. 2023Imitation Learning from Observation with Automatic Discount Scheduling
  6. 2022Imitating, Fast and Slow: Robust learning from demonstrations via decision-time planning
  7. 2021Imitation Learning by Reinforcement Learning
  8. 2025Reinforcement Learning via Implicit Imitation Guidance