mlsanityData to 14 Oct 2026

Uncertainty-Based Smooth Policy Regularisation for Reinforcement Learning with Few Demonstrations

arXiv 2509.15981 ↗First posted 19 Sep 2025cs.LG cs.AI cs.ROcited by 0

On the map: Deep reinforcement learning methods → RL for robot locomotion and manipulation

Abstract (arXiv)

In reinforcement learning with sparse rewards, demonstrations can accelerate learning, but determining when to imitate them remains challenging. We propose Smooth Policy Regularisation from Demonstrations (SPReD), a framework that addresses the fundamental question: when should an agent imitate a demonstration versus follow its own policy? SPReD uses ensemble methods to explicitly model Q-value distributions for both demonstration and policy actions, quantifying uncertainty for comparisons. We develop two complementary uncertainty-aware methods: a probabilistic approach estimating the likelihood of demonstration superiority, and an advantage-based approach scaling imitation by statistical significance. Unlike prevailing methods (e.g. Q-filter) that make binary imitation decisions, SPReD applies continuous, uncertainty-proportional regularisation weights, reducing gradient variance during training. Despite its computational simplicity, SPReD achieves remarkable gains in experiments across eight robotics tasks, outperforming existing approaches by up to a factor of 14 in complex tasks while maintaining robustness to demonstration quality and quantity. Our code is available at https://github.com/YujieZhu7/SPReD.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods. Builds on 38 of its 63 references that are in the map. First posted on arXiv 19 Sep 2025.

Builds on

38 of 63 references.

References · 38

  1. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  2. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  4. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  7. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  8. 2017Hindsight Experience Replaycited by 1,136
  9. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  10. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  11. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  12. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
26 more
  1. 2016Deep Exploration via Bootstrapped DQNcited by 527
  2. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 371
  3. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  4. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 340
  5. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 319
  6. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 302
  7. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  8. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 260
  9. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 214
  10. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learningcited by 126
  11. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 104
  12. 2017Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Predictioncited by 93
  13. 2024From Imitation to Refinement -- Residual RL for Precise Assemblycited by 90
  14. 2019Better Exploration with Optimistic Actor-Criticcited by 88
  15. 2017The Uncertainty Bellman Equation and Explorationcited by 82
  16. 2016Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Criticcited by 81
  17. 2018Reinforcement Learning from Imperfect Demonstrationscited by 75
  18. 2019Estimating Risk and Uncertainty in Deep Reinforcement Learningcited by 70
  19. 2022An Improved Analysis of (Variance-Reduced) Policy Gradient and Natural Policy Gradient Methodscited by 68
  20. 2022A Review of Uncertainty for Deep Reinforcement Learningcited by 40
  21. 2019Reinforcement Learning from Imperfect Demonstrations under Soft Expert Guidancecited by 25
  22. 2022Improving TD3-BC: Relaxed Policy Constraint for Offline Learning and Stable Online Fine-Tuningcited by 20
  23. 2018Learning with Training Wheels: Speeding up Training with a Simple Controller for Deep Reinforcement Learningcited by 17
  24. 2019Curiosity-Driven Multi-Criteria Hindsight Experience Replaycited by 11
  25. 2018Active Deep Q-learning with Demonstrationcited by 10
  26. 2022Bayesian Q-learning With Imperfect Expert Demonstrationscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Demonstration-Regularized RL
  2. 2022Robust Learning from Observation with Model Misspecification
  3. 2022Robust Imitation of a Few Demonstrations with a Backwards Model
  4. 2022Autonomous Assessment of Demonstration Sufficiency via Bayesian Inverse Reinforcement Learning
  5. 2021Confidence-Aware Imitation Learning from Demonstrations with Varying Optimality
  6. 2019VILD: Variational Imitation Learning with Diverse-quality Demonstrations
  7. 2024Uncertainty-Aware Deployment of Pre-trained Language-Conditioned Imitation Learning Policies
  8. 2021Learning from Imperfect Demonstrations from Agents with Varying Dynamics