mlsanityData to 7 Oct 2026

Self-Imitation Learning via Generalized Lower Bound Q-learning

arXiv 2006.07442 ↗First posted 12 Jun 2020cs.LG stat.MLcited by 13

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Self-imitation learning motivated by lower-bound Q-learning is a novel and effective approach for off-policy learning. In this work, we propose a n-step lower bound which generalizes the original return-based lower-bound Q-learning, and introduce a new family of self-imitation learning algorithms. To provide a formal motivation for the potential performance gains provided by self-imitation learning, we show that n-step lower bound Q-learning achieves a trade-off between fixed point bias and contraction rate, drawing close connections to the popular uncorrected n-step Q-learning. We finally show that n-step lower bound Q-learning is a more robust alternative to return-based self-imitation learning and uncorrected n-step, over a wide range of continuous control benchmark tasks.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 33 of its 52 references that are in the map. First posted on arXiv 12 Jun 2020.

Builds on

33 of 52 references.

Closest ancestors · 2

Same direction, newest first

  1. 2018Self-Imitation Learning14 Jun
  2. 2018Learning Self-Imitating Diverse Policies25 May

Other references · 31

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2015Continuous control with deep reinforcement learningcited by 3,279
  5. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  6. 2015Trust Region Policy Optimizationcited by 2,503
  7. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  8. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  9. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  10. 2015Prioritized Experience Replaycited by 1,070
  11. 2018DeepMind Control Suitecited by 935
  12. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
19 more
  1. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  2. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  3. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 285
  4. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
  5. 2018Distributed Prioritized Experience Replaycited by 260
  6. 2018Distributed Distributional Deterministic Policy Gradientscited by 195
  7. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 178
  8. 2015Doubly Robust Policy Evaluation and Optimizationcited by 158
  9. 2017Neural Episodic Controlcited by 148
  10. 2018More Robust Doubly Robust Off-policy Evaluationcited by 141
  11. 2015Taming the Noise in Reinforcement Learning via Soft Updatescited by 125
  12. 2020Maxmin Q-learning: Controlling the Estimation Bias of Q-learningcited by 117
  13. 2016Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learningcited by 106
  14. 2016Model-Free Episodic Controlcited by 100
  15. 2016Q(λ) with Off-Policy Correctionscited by 48
  16. 2016Learning to Play in a Day: Faster Deep Reinforcement Learning by Optimality Tighteningcited by 21
  17. 2017Multi-step Off-policy Learning Without Importance Sampling Ratioscited by 19
  18. 2019Adaptive Trade-Offs in Off-Policy Learningcited by 19
  19. 2019Memory Based Trajectory-conditioned Policies for Learning from Sparse Rewardscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 2; 2022: 4; 2023: 2; 2024: 1; 2025: 2; 2026: 1.

13 papers.

Sort:

By area:

  1. 2021Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domaincited by 57
  2. 2021Offline Reinforcement Learning with Value-based Episodic Memorycited by 36
  3. 2020Episodic Self-Imitation Learning with Hindsightcited by 10
  4. 2025Learn the Ropes, Then Trust the Wins: Self-imitation with Progressive Exploration for Agentic Reinforcement Learningcited by 10
  5. 2022Learning Action Translator for Meta Reinforcement Learning on Sparse-Reward Taskscited by 5
  6. 2023Visual Hindsight Self-Imitation Learning for Interactive Navigationcited by 5
  7. 2022Self-Imitation Learning from Demonstrationscited by 4
  8. 2024Boosting Soft Q-Learning by Boundingcited by 4
  9. 2025GCHR : Goal-Conditioned Hindsight Regularization for Sample-Efficient Reinforcement Learningcited by 2
  10. 2022Accelerating Self-Imitation Learning from Demonstrations via Policy Constraints and Q-Ensemblecited by 1
  11. 2023Efficient Sparse-Reward Goal-Conditioned Reinforcement Learning with a High Replay Ratio and Regularizationcited by 1
  12. 2022Multi-Step First: A Lightweight Deep Reinforcement Learning Strategy for Robust Continuous Control with Partial Observabilitycited by 0
1 more
  1. 2026Match or Replay: Self Imitating Proximal Policy Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Inverse Q-Learning Done Right: Offline Imitation Learning in Qπ-Realizable MDPs
  2. 2023Imitation Learning from Nonlinear MPC via the Exact Q-Loss and its Gauss-Newton Approximation
  3. 2021Quantile Filtered Imitation Learning
  4. 2021Imitation Learning by Reinforcement Learning
  5. 2022TaSIL: Taylor Series Imitation Learning
  6. 2023Residual Q-Learning: Offline and Online Policy Customization without Value
  7. 2021IQ-Learn: Inverse soft-Q Learning for Imitation
  8. 2021On the Sample Complexity of Stability Constrained Imitation Learning