mlsanityData to 7 Oct 2026

Boosting Reinforcement Learning with Strongly Delayed Feedback Through Auxiliary Short Delays

arXiv 2402.03141 ↗First posted 5 Feb 2024cs.LG cs.AI cs.SYcited by 10

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning (RL) is challenging in the common case of delays between events and their sensory perceptions. State-of-the-art (SOTA) state augmentation techniques either suffer from state space explosion or performance degeneration in stochastic environments. To address these challenges, we present a novel Auxiliary-Delayed Reinforcement Learning (AD-RL) method that leverages auxiliary tasks involving short delays to accelerate RL with long delays, without compromising performance in stochastic environments. Specifically, AD-RL learns a value function for short delays and uses bootstrapping and policy improvement techniques to adjust it for long delays. We theoretically show that this can greatly reduce the sample complexity. On deterministic and stochastic benchmarks, our method significantly outperforms the SOTAs in both sample efficiency and policy performance. Code is available at https://github.com/QingyuanWuNothing/AD-RL.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 19 of its 53 references that are in the map. First posted on arXiv 5 Feb 2024.

Builds on

19 of 53 references.

References · 19

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  3. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  4. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  5. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  6. 2017Control of a Quadrotor with Reinforcement Learningcited by 75
  7. 2020Reinforcement Learning with Random Delayscited by 56
  8. 2022Enforcing Hard Constraints with Soft Barriers: Safe Reinforcement Learning in Unknown Stochastic Environmentscited by 54
  9. 2020Delay-Aware Model-Based Reinforcement Learning for Continuous Controlcited by 33
  10. 2018Setting up a Reinforcement Learning Task with a Real-World Robotcited by 32
  11. 2021Acting in Delayed Environments with Non-Stationary Markov Policiescited by 28
  12. 2021Off-Policy Reinforcement Learning with Delayed Rewardscited by 25
7 more
  1. 2018At Human Speed: Deep Reinforcement Learning with Action Delaycited by 24
  2. 2019Learning Belief Representations for Imitation Learning in POMDPscited by 23
  3. 2021Revisiting State Augmentation methods for Reinforcement Learning with Stochastic Delayscited by 20
  4. 2022Delayed Reinforcement Learning by Imitationcited by 15
  5. 2022Joint Differentiable Optimization and Verification for Certified Reinforcement Learningcited by 11
  6. 2023State-Wise Safe Reinforcement Learning With Pixel Observationscited by 9
  7. 2021Learning-based Framework for Sensor Fault-Tolerant Building HVAC Control with Model-assisted Learningcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 5.

10 papers.

  1. 2024Enhancing Inverse Reinforcement Learning through Encoding Dynamic Information in Reward Shapingcited by 3
  2. 2025Model-Based Reinforcement Learning under Random Observation Delayscited by 2
  3. 2025Belief-Based Offline Reinforcement Learning for Delay-Robust Policy Optimizationcited by 1
  4. 2025Reinforcement Learning via Conservative Agent for Environments with Random Delayscited by 1
  5. 2025Learning Multi-Timescale Interventions under Safety and Resource Constraintscited by 0
  6. 2026State-Action Inpainting Diffuser for Continuous Control with Delaycited by 0
  7. 2026Minimax Optimal Strategy for Delayed Observations in Online Reinforcement Learningcited by 0
  8. 2026Delayed homomorphic reinforcement learning for environments with delayed feedbackcited by 0
  9. 2026Structural Equivalence and Learning Dynamics in Delayed MARLcited by 0
  10. 2026LangLoc: "Tell Me What You See"cited by 0