mlsanityData to 7 Oct 2026

Accelerating Self-Imitation Learning from Demonstrations via Policy Constraints and Q-Ensemble

arXiv 2212.03562 ↗First posted 7 Dec 2022cs.LGcited by 1

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Deep reinforcement learning (DRL) provides a new way to generate robot control policy. However, the process of training control policy requires lengthy exploration, resulting in a low sample efficiency of reinforcement learning (RL) in real-world tasks. Both imitation learning (IL) and learning from demonstrations (LfD) improve the training process by using expert demonstrations, but imperfect expert demonstrations can mislead policy improvement. Offline to Online reinforcement learning requires a lot of offline data to initialize the policy, and distribution shift can easily lead to performance degradation during online fine-tuning. To solve the above problems, we propose a learning from demonstrations method named A-SILfD, which treats expert demonstrations as the agent's successful experiences and uses experiences to constrain policy improvement. Furthermore, we prevent performance degradation due to large estimation errors in the Q-function by the ensemble Q-functions. Our experiments show that A-SILfD can significantly improve sample efficiency using a small number of different quality expert demonstrations. In four Mujoco continuous control tasks, A-SILfD can significantly outperform baseline methods after 150,000 steps of online training and is not misled by imperfect expert demonstrations during training.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 32 of its 57 references that are in the map. First posted on arXiv 7 Dec 2022.

Builds on

32 of 57 references.

Closest ancestors · 8

Same direction, newest first

  1. 2022Self-Imitation Learning from Demonstrations21 Mar
  2. 2021Off-Policy Imitation Learning from Observations25 Feb
  3. 2020Self-Imitation Learning via Generalized Lower Bound Q-learning12 Jun
  4. 2019A Divergence Minimization Perspective on Imitation Learning Methods6 Nov
  5. 2019Imitation Learning from Imperfect Demonstration27 Jan
  6. 2018Generative Adversarial Self-Imitation Learning3 Dec
  7. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning9 Sep
  8. 2018Self-Imitation Learning14 Jun

Other references · 24

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2015Trust Region Policy Optimizationcited by 2,503
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  5. 2016Generative Adversarial Imitation Learningcited by 1,576
  6. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  8. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  9. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  10. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  11. 2018Learning Dexterous In-Hand Manipulationcited by 643
  12. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
12 more
  1. 2020Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Surveycited by 388
  2. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  3. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 297
  4. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  5. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 211
  6. 2020f-IRL: Inverse Reinforcement Learning via State Marginal Matchingcited by 67
  7. 2022Reinforcement Learning with Sparse Rewards using Guidance from Offline Demonstrationcited by 54
  8. 2022Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learningcited by 34
  9. 2022Hierarchical Reinforcement Learning for Precise Soccer Shooting Skills using a Quadrupedal Robotcited by 32
  10. 2020Supervised Learning Achieves Human-Level Performance in MOBA Games: A Case Study of Honor of Kingscited by 18
  11. 2017Pre-training Neural Networks with Human Demonstrations for Deep Reinforcement Learningcited by 15
  12. 2022MOORe: Model-based Offline-to-Online Reinforcement Learningcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Repetition as Reinforcement: Enhancing Sample Efficiency via Instant Episode Repetition in Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Imitation Bootstrapped Reinforcement Learning
  2. 2019SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards
  3. 2025Enhancing Online Reinforcement Learning with Meta-Learned Objective from Offline Data
  4. 2021Efficiently Training On-Policy Actor-Critic Networks in Robotic Deep Reinforcement Learning with Demonstration-like Sampled Exploration
  5. 2022Divide & Conquer Imitation Learning
  6. 2026Test-Time Gradient Guidance of Flow Policies in Reinforcement Learning
  7. 2021Learning from demonstrations with SACR2: Soft Actor-Critic with Reward Relabeling
  8. 2022Efficient Reinforcement Learning from Demonstration Using Local Ensemble and Reparameterization with Split and Merge of Expert Policies