mlsanityData to 7 Oct 2026

Boosting Offline Reinforcement Learning with Residual Generative Modeling

arXiv 2106.10411 ↗First posted 19 Jun 2021cs.LG cs.AIcited by 7

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) tries to learn the near-optimal policy with recorded offline experience without online exploration. Current offline RL research includes: 1) generative modeling, i.e., approximating a policy using fixed data; and 2) learning the state-action value function. While most research focuses on the state-action function part through reducing the bootstrapping error in value function approximation induced by the distribution shift of training data, the effects of error propagation in generative modeling have been neglected. In this paper, we analyze the error in generative modeling. We propose AQL (action-conditioned Q-learning), a residual generative model to reduce policy approximation error for offline RL. We show that our method can learn more accurate policy approximations in different benchmark datasets. In addition, we show that the proposed offline RL method can learn more competitive AI agents in complex control tasks under the multiplayer online battle arena (MOBA) game Honor of Kings.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 19 of its 29 references that are in the map. First posted on arXiv 19 Jun 2021.

Builds on

19 of 29 references.

Closest ancestors · 4

Same direction, newest first

  1. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  4. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 15

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  5. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  6. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  7. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  8. 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 413
  9. 2019Challenges of Real-World Reinforcement Learningcited by 282
  10. 2019Mastering Complex Control in MOBA Games with Deep Reinforcement Learningcited by 150
  11. 2020Towards Playing Full MOBA Games with Deep Reinforcement Learningcited by 78
  12. 2017Learning Deep ResNet Blocks Sequentially using Boosting Theorycited by 32
3 more
  1. 2020Supervised Learning Achieves Human-Level Performance in MOBA Games: A Case Study of Honor of Kingscited by 18
  2. 2018Are ResNets Provably Better than Linear Predictors?cited by 12
  3. 2020Which Heroes to Pick? Learning to Draft in MOBA Games with Neural Networks and Tree Searchcited by 10

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 1; 2023: 0; 2024: 3; 2025: 1.

7 papers.

  1. 2022Honor of Kings Arena: an Environment for Generalization in Competitive Reinforcement Learningcited by 27
  2. 2024Decision Mamba: A Multi-Grained State Space Model with Self-Evolution Regularization for Offline RLcited by 15
  3. 2024Hokoff: Real Game Dataset from Honor of Kings and its Offline Reinforcement Learning Benchmarkscited by 10
  4. 2021Learning Diverse Policies in MOBA Games via Macro-Goalscited by 7
  5. 2024Preferred-Action-Optimized Diffusion Policies for Offline Reinforcement Learningcited by 3
  6. 2025DR-SAC: Distributionally Robust Soft Actor-Critic for Reinforcement Learning under Uncertaintycited by 3
  7. 2021Compressive Features in Offline Reinforcement Learning for Recommender Systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modeling
  2. 2021A Minimalist Approach to Offline Reinforcement Learning
  3. 2025Unsupervised Data Generation for Offline Reinforcement Learning: A Perspective from Model
  4. 2021Offline Reinforcement Learning with Implicit Q-Learning
  5. 2021Offline RL Without Off-Policy Evaluation
  6. 2021The Least Restriction for Offline Reinforcement Learning
  7. 2020MOReL : Model-Based Offline Reinforcement Learning
  8. 2021Offline Reinforcement Learning with Reverse Model-based Imagination