mlsanityData to 14 Oct 2026

Variational Delayed Policy Optimization

arXiv 2405.14226 ↗First posted 23 May 2024cs.LG cs.AIcited by 9

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

In environments with delayed observation, state augmentation by including actions within the delay window is adopted to retrieve Markovian property to enable reinforcement learning (RL). However, state-of-the-art (SOTA) RL techniques with Temporal-Difference (TD) learning frameworks often suffer from learning inefficiency, due to the significant expansion of the augmented state space with the delay. To improve learning efficiency without sacrificing performance, this work introduces a novel framework called Variational Delayed Policy Optimization (VDPO), which reformulates delayed RL as a variational inference problem. This problem is further modelled as a two-step iterative optimization problem, where the first step is TD learning in the delay-free environment with a small state space, and the second step is behaviour cloning which can be addressed much more efficiently than TD learning. We not only provide a theoretical analysis of VDPO in terms of sample complexity and performance, but also empirically demonstrate that VDPO can achieve consistent performance with SOTA methods, with a significant enhancement of sample efficiency (approximately 50% less amount of samples) in the MuJoCo benchmark.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 25 of its 44 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

25 of 44 references.

References · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  4. 2016Generative Adversarial Imitation Learningcited by 1,597
  5. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  6. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  7. 2015Deep Learning and the Information Bottleneck Principlecited by 802
  8. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  9. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 490
  10. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  11. 2018Maximum a Posteriori Policy Optimisationcited by 328
  12. 2017Control of a Quadrotor with Reinforcement Learningcited by 92
13 more
  1. 2022Constrained Variational Policy Optimization for Safe Reinforcement Learningcited by 71
  2. 2022Enforcing Hard Constraints with Soft Barriers: Safe Reinforcement Learning in Unknown Stochastic Environmentscited by 60
  3. 2020Reinforcement Learning with Random Delayscited by 59
  4. 2020Delay-Aware Model-Based Reinforcement Learning for Continuous Controlcited by 34
  5. 2018Setting up a Reinforcement Learning Task with a Real-World Robotcited by 33
  6. 2018Relative Entropy Regularized Policy Iterationcited by 32
  7. 2018At Human Speed: Deep Reinforcement Learning with Action Delaycited by 26
  8. 2018VIREL: A Variational Inference Framework for Reinforcement Learningcited by 22
  9. 2021Revisiting State Augmentation methods for Reinforcement Learning with Stochastic Delayscited by 21
  10. 2022Delayed Reinforcement Learning by Imitationcited by 16
  11. 2022Joint Differentiable Optimization and Verification for Certified Reinforcement Learningcited by 11
  12. 2023State-Wise Safe Reinforcement Learning With Pixel Observationscited by 9
  13. 2023Delays in Reinforcement Learningcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 4; 2026: 3.

9 papers.

  1. 2025Advancing Autonomous VLM Agents via Variational Subgoal-Conditioned Reinforcement Learningcited by 5
  2. 2024Enhancing Inverse Reinforcement Learning through Encoding Dynamic Information in Reward Shapingcited by 3
  3. 2025Model-Based Reinforcement Learning under Random Observation Delayscited by 3
  4. 2025Belief-Based Offline Reinforcement Learning for Delay-Robust Policy Optimizationcited by 1
  5. 2025Reinforcement Learning via Conservative Agent for Environments with Random Delayscited by 1
  6. 2026Reinforcement Learning for Control Systems with Time Delays: A Comprehensive Surveycited by 1
  7. 2024Inverse Delayed Reinforcement Learningcited by 0
  8. 2026State-Action Inpainting Diffuser for Continuous Control with Delaycited by 0
  9. 2026Delayed homomorphic reinforcement learning for environments with delayed feedbackcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Boosting Reinforcement Learning with Strongly Delayed Feedback Through Auxiliary Short Delays
  2. 2025Deterministic Policy Gradient for Reinforcement Learning with Continuous Time and State
  3. 2024Accelerating Proximal Policy Optimization Learning Using Task Prediction for Solving Environments with Delayed Rewards
  4. 2025Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning
  5. 2024Statistical Inference for Policy Evaluation with Temporal Difference Learning
  6. 2022Offline Reinforcement Learning with Closed-Form Policy Improvement Operators
  7. 2026Dynamical Priors as a Training Objective in Reinforcement Learning
  8. 2026Diffusion-Guided Uncertainty-Aware Delayed Policy Optimization