# Revisiting State Augmentation methods for Reinforcement Learning with Stochastic Delays

arXiv: https://arxiv.org/abs/2108.07555 · HTML: https://mlsanity.com/paper/2108.07555/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 38 references that are in the map.
- First posted on arXiv 17 Aug 2021.

## Builds on (most cited first, 8 of 8)

- [Deep Reinforcement Learning that Matters](https://mlsanity.com/paper/1709.06560/) (2017)
- [Reinforcement Learning with Random Delays](https://mlsanity.com/paper/2010.02966/) (2020)
- [Real-Time Reinforcement Learning](https://mlsanity.com/paper/1911.04448/) (2019)
- [Delay-Aware Model-Based Reinforcement Learning for Continuous Control](https://mlsanity.com/paper/2005.05440/) (2020)
- [Thinking While Moving: Deep Reinforcement Learning with Concurrent Control](https://mlsanity.com/paper/2004.06089/) (2020)
- [Acting in Delayed Environments with Non-Stationary Markov Policies](https://mlsanity.com/paper/2101.11992/) (2021)
- [Blind Decision Making: Reinforcement Learning with Delayed Observations](https://mlsanity.com/paper/2011.07715/) (2020)
- [On Unbounded Delays in Asynchronous Parallel Fixed-Point Algorithms](https://mlsanity.com/paper/1609.04746/) (2016)

## Built on by (most cited first, 21 of 21)

- [A Survey of Sim-to-Real Methods in RL: Progress, Prospects and Challenges with Foundation Models](https://mlsanity.com/paper/2502.13187/) (2025)
- [Learning Long-Term Reward Redistribution via Randomized Return Decomposition](https://mlsanity.com/paper/2111.13485/) (2021)
- [Decision Transformer under Random Frame Dropping](https://mlsanity.com/paper/2303.03391/) (2023)
- [Boosting Reinforcement Learning with Strongly Delayed Feedback Through Auxiliary Short Delays](https://mlsanity.com/paper/2402.03141/) (2024)
- [Variational Delayed Policy Optimization](https://mlsanity.com/paper/2405.14226/) (2024)
- [DiAReL: Reinforcement Learning with Disturbance Awareness for Robust Sim2Real Policy Transfer in Robot Control](https://mlsanity.com/paper/2306.09010/) (2023)
- [DEER: A Delay-Resilient Framework for Reinforcement Learning with Variable Delays](https://mlsanity.com/paper/2406.03102/) (2024)
- [Enabling Realtime Reinforcement Learning at Scale with Staggered Asynchronous Inference](https://mlsanity.com/paper/2412.14355/) (2024)
- [Multi-Timescale Control and Communications with Deep Reinforcement Learning -- Part I: Communication-Aware Vehicle Control](https://mlsanity.com/paper/2311.11281/) (2023)
- [Adaptive PD Control using Deep Reinforcement Learning for Local-Remote Teleoperation with Stochastic Time Delays](https://mlsanity.com/paper/2305.16979/) (2023)
- [Reinforcement Learning for Control Systems with Time Delays: A Comprehensive Survey](https://mlsanity.com/paper/2602.00399/) (2026)
- [A Learning Based Framework for Handling Uncertain Lead Times in Multi-Product Inventory Management](https://mlsanity.com/paper/2203.00885/) (2022)
- [Trajectory Planning for Teleoperated Space Manipulators Using Deep Reinforcement Learning](https://mlsanity.com/paper/2408.05460/) (2024)
- [State-Action Inpainting Diffuser for Continuous Control with Delay](https://mlsanity.com/paper/2603.01553/) (2026)
- [Delay-Empowered Causal Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2605.12261/) (2026)
- [Residual Reinforcement Learning for Robot Teleoperation under Stochastic Delays](https://mlsanity.com/paper/2605.15480/) (2026)
- [Decoupled Delay Compensation: Enhancing Pre-trained MARL Policies via Learned Dynamics Filtering](https://mlsanity.com/paper/2605.26286/) (2026)
- [OnDeFog: Online Decision Transformer under Frame Dropping](https://mlsanity.com/paper/2606.19721/) (2026)
- [Hierarchical Reinforcement Learning for Neural Network Compression (HiReLC): Pruning and Quantization](https://mlsanity.com/paper/2606.26002/) (2026)
- [LangLoc: "Tell Me What You See"](https://mlsanity.com/paper/2607.05077/) (2026)
- [RAPAC-DP: Response-Aligned Pending-Action Compensation for Diffusion Policies under Delayed Execution](https://mlsanity.com/paper/2608.15924/) (2026)

## Abstract (arXiv)

Several real-world scenarios, such as remote control and sensing, are comprised of action and observation delays. The presence of delays degrades the performance of reinforcement learning (RL) algorithms, often to such an extent that algorithms fail to learn anything substantial. This paper formally describes the notion of Markov Decision Processes (MDPs) with stochastic delays and shows that delayed MDPs can be transformed into equivalent standard MDPs (without delays) with significantly simplified cost structure. We employ this equivalence to derive a model-free Delay-Resolved RL framework and show that even a simple RL algorithm built upon this framework achieves near-optimal rewards in environments with stochastic delays in actions and observations. The delay-resolved deep Q-network (DRDQN) algorithm is bench-marked on a variety of environments comprising of multi-step and stochastic delays and results in better performance, both in terms of achieving near-optimal rewards and minimizing the computational overhead thereof, with respect to the currently established algorithms.