mlsanityData to 14 Oct 2026

Off-Policy Reinforcement Learning with Delayed Rewards

arXiv 2106.11854 ↗First posted 22 Jun 2021cs.LGcited by 25

Not on the map. Why?

Abstract (arXiv)

We study deep reinforcement learning (RL) algorithms with delayed rewards. In many real-world tasks, instant rewards are often not readily accessible or even defined immediately after the agent performs actions. In this work, we first formally define the environment with delayed rewards and discuss the challenges raised due to the non-Markovian nature of such environments. Then, we introduce a general off-policy RL framework with a new Q-function formulation that can handle the delayed rewards with theoretical convergence guarantees. For practical tasks with high dimensional state spaces, we further introduce the HC-decomposition rule of the Q-function in our framework which naturally leads to an approximation scheme that helps boost the training efficiency and stability. We finally conduct extensive experiments to demonstrate the superior performance of our algorithms over the existing work and their variants.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Builds on 22 of its 43 references that are in the map. First posted on arXiv 22 Jun 2021.

Builds on

22 of 43 references.

References · 22

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  5. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  6. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  7. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  8. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  9. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  10. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  11. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 371
  12. 2017Molecular De Novo Design through Deep Reinforcement Learningcited by 267
10 more
  1. 2018RUDDER: Return Decomposition for Delayed Rewardscited by 173
  2. 2018On Learning Intrinsic Rewards for Policy Gradient Methodscited by 102
  3. 2017A Benchmark Environment Motivated by Industrial Control Problemscited by 31
  4. 2021Acting in Delayed Environments with Non-Stationary Markov Policiescited by 31
  5. 2020Learning Guidance Rewards with Trajectory-space Smoothingcited by 30
  6. 2018Learning Self-Imitating Diverse Policiescited by 29
  7. 2019Sequence Modeling of Temporal Credit Assignment for Episodic Reinforcement Learningcited by 27
  8. 2021Self-Imitation Learning by Planningcited by 26
  9. 2018Generative Adversarial Self-Imitation Learningcited by 25
  10. 2020Reward Propagation Using Graph Convolutional Networkscited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 2; 2023: 5; 2024: 8; 2025: 3; 2026: 6.

25 papers.

Sort:

By area:

  1. 2024Dense Reward for Free in Reinforcement Learning from Human Feedbackcited by 78
  2. 2021Learning Long-Term Reward Redistribution via Randomized Return Decompositioncited by 33
  3. 2024Latent Reward: LLM-Empowered Credit Assignment in Episodic Reinforcement Learningcited by 23
  4. 2025Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learningcited by 23
  5. 2023Interpretable Reward Redistribution in Reinforcement Learning: A Causal Approachcited by 19
  6. 2022Provably Efficient Offline Reinforcement Learning with Trajectory-Wise Rewardcited by 13
  7. 2024Boosting Reinforcement Learning with Strongly Delayed Feedback Through Auxiliary Short Delayscited by 10
  8. 2023Off-Policy Evaluation for Human Feedbackcited by 6
  9. 2023STAS: Spatial-Temporal Return Decomposition for Multi-agent Reinforcement Learningcited by 5
  10. 2023Posterior Sampling with Delayed Feedback for Reinforcement Learning with Linear Function Approximationcited by 3
  11. 2023Episodic Return Decomposition by Difference of Implicitly Assigned Sub-Trajectory Rewardcited by 2
  12. 2024Highway Reinforcement Learningcited by 2
13 more
  1. 2024Beyond Simple Sum of Delayed Rewards: Non-Markovian Reward Modeling for Reinforcement Learningcited by 2
  2. 2022Off-Beat Multi-Agent Reinforcement Learningcited by 1
  3. 2024Agent-Temporal Credit Assignment for Optimal Policy Preservation in Sparse Multi-Agent Reinforcement Learningcited by 1
  4. 2025Belief-Based Offline Reinforcement Learning for Delay-Robust Policy Optimizationcited by 1
  5. 2025Synthetic POMDPs to Challenge Memory-Augmented RL: Memory Demand Structure Modelingcited by 1
  6. 2024Reinforcement Learning from Bagged Rewardcited by 0
  7. 2024Inverse Delayed Reinforcement Learningcited by 0
  8. 2026PYTHALAB-MERA: Validation-Grounded Memory, Retrieval, and Acceptance Control for Frozen-LLM Coding Agentscited by 0
  9. 2026Delay-Empowered Causal Hierarchical Reinforcement Learningcited by 0
  10. 2026Retroactive Advantage Correction: Closed-Form V-Trace Bias Correction for Delay-Aware RLHFcited by 0
  11. 2026Explore Beyond the Boundary Using Entropic Informationcited by 0
  12. 2026Learning Reliable Parking Policies via Offline Reinforcement Learning with Quantized Action Representationscited by 0
  13. 2026SpikeCredit: Temporal Credit Carrier for Reinforcement Learning with Sparse Rewardscited by 0