mlsanityData to 13 Oct 2026

On Pathologies in KL-Regularized Reinforcement Learning from Expert Demonstrations

arXiv 2212.13936 ↗First posted 28 Dec 2022cs.LG cs.AI stat.MEcited by 18

Not on the map. Why?

Abstract (arXiv)

KL-regularized reinforcement learning from expert demonstrations has proved successful in improving the sample efficiency of deep reinforcement learning algorithms, allowing them to be applied to challenging physical real-world tasks. However, we show that KL-regularized reinforcement learning with behavioral reference policies derived from expert demonstrations can suffer from pathological training dynamics that can lead to slow, unstable, and suboptimal online learning. We show empirically that the pathology occurs for commonly chosen behavioral policy classes and demonstrate its impact on sample efficiency and online policy performance. Finally, we show that the pathology can be remedied by non-parametric behavioral reference policies and that this allows KL-regularized reinforcement learning to significantly outperform state-of-the-art approaches on a variety of challenging locomotion and dexterous hand manipulation tasks.

Key facts. Cited by 18 papers in the map (data through 13 Oct 2026). Builds on 25 of its 52 references that are in the map. First posted on arXiv 28 Dec 2022.

Builds on

25 of 52 references.

References · 25

  1. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,902
  2. 2015Continuous control with deep reinforcement learningcited by 3,290
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,221
  4. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,185
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  6. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 767
  7. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 687
  8. 2019Behavior Regularized Offline Reinforcement Learningcited by 657
  9. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 654
  10. 2020MOPO: Model-based Offline Policy Optimizationcited by 632
  11. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 584
  12. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 477
13 more
  1. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 366
  2. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 325
  3. 2019Challenges of Real-World Reinforcement Learningcited by 285
  4. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learningcited by 214
  5. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 172
  6. 2019Exact Gaussian Processes on a Million Data Pointscited by 118
  7. 2018Reinforcement Learning from Imperfect Demonstrationscited by 74
  8. 2019Information asymmetry in KL-regularized RLcited by 53
  9. 2021Augmented World Models Facilitate Zero-Shot Dynamics Generalization From a Single Offline Environmentcited by 41
  10. 2019Radial Bayesian Neural Networks: Beyond Discrete Support In Large-Scale Bayesian Deep Learningcited by 37
  11. 2019Learning to Score Behaviors for Guided Policy Optimizationcited by 34
  12. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RLcited by 24
  13. 2021Outcome-Driven Reinforcement Learning via Variational Inferencecited by 17

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 8; 2024: 6; 2025: 2; 2026: 2.

18 papers.

Sort:

By area:

  1. 2023Efficient Online Reinforcement Learning with Offline Datacited by 334
  2. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 104
  3. 2024From Imitation to Refinement -- Residual RL for Precise Assemblycited by 81
  4. 2023Imitation Bootstrapped Reinforcement Learningcited by 55
  5. 2023Tractable Function-Space Variational Inference in Bayesian Neural Networkscited by 41
  6. 2023Learning and Adapting Agile Locomotion Skills by Transferring Experiencecited by 25
  7. 2023Demonstration-Guided Reinforcement Learning with Efficient Exploration for Task Automation of Surgical Robotcited by 21
  8. 2023Coherent Soft Imitation Learningcited by 21
  9. 2024Context-Guided Diffusion for Out-of-Distribution Molecular and Protein Designcited by 11
  10. 2024DEER: A Delay-Resilient Framework for Reinforcement Learning with Variable Delayscited by 6
  11. 2024FOSP: Fine-tuning Offline Safe Policy through World Modelscited by 3
  12. 2024Towards Fast Safe Online Reinforcement Learning via Policy Finetuningcited by 3
6 more
  1. 2024Goal-Reaching Policy Learning from Non-Expert Observations via Effective Subgoal Guidancecited by 2
  2. 2025On the Effect of Regularization in Policy Mirror Descentcited by 1
  3. 2026XQCfD: Accelerating Fast Actor-Critic Algorithms with Prior Data and Prior Policiescited by 1
  4. 2023Mitigating Estimation Errors by Twin TD-Regularized Actor and Critic for Deep Reinforcement Learningcited by 0
  5. 2025Learning from Suboptimal Data in Continuous Control via Auto-Regressive Soft Q-Networkcited by 0
  6. 2026Well-Posed KL-Regularized Control via Wasserstein and Kalman-Wasserstein KL Divergencescited by 0