mlsanityData to 14 Oct 2026

Some Supervision Required: Incorporating Oracle Policies in Reinforcement Learning via Epistemic Uncertainty Metrics

arXiv 2208.10533 ↗First posted 22 Aug 2022cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

An inherent problem of reinforcement learning is performing exploration of an environment through random actions, of which a large portion can be unproductive. Instead, exploration can be improved by initializing the learning policy with an existing (previously learned or hard-coded) oracle policy, offline data, or demonstrations. In the case of using an oracle policy, it can be unclear how best to incorporate the oracle policy's experience into the learning policy in a way that maximizes learning sample efficiency. In this paper, we propose a method termed Critic Confidence Guided Exploration (CCGE) for incorporating such an oracle policy into standard actor-critic reinforcement learning algorithms. More specifically, CCGE takes in the oracle policy's actions as suggestions and incorporates this information into the learning scheme when uncertainty is high, while ignoring it when the uncertainty is low. CCGE is agnostic to methods of estimating uncertainty, and we show that it is equally effective with two different techniques. Empirically, we evaluate the effect of CCGE on various benchmark reinforcement learning tasks, and show that this idea can lead to improved sample efficiency and final performance. Furthermore, when evaluated on sparse reward environments, CCGE is able to perform competitively against adjacent algorithms that also leverage an oracle policy. Our experiments show that it is possible to utilize uncertainty as a heuristic to guide exploration using an oracle in reinforcement learning. We expect that this will inspire more research in this direction, where various heuristics are used to determine the direction of guidance provided to learning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 51 of its 88 references that are in the map. First posted on arXiv 22 Aug 2022.

Builds on

51 of 88 references.

References · 51

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  5. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  6. 2015Continuous control with deep reinforcement learningcited by 3,407
  7. 2015Variational Inference with Normalizing Flowscited by 2,170
  8. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  9. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  10. 2016OpenAI Gymcited by 1,736
  11. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  12. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
38 more
  1. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  2. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  4. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  5. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  6. 2020Mastering Atari with Discrete World Modelscited by 1,007
  7. 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 913
  8. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  9. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  10. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  11. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  12. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  13. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  14. 2018Predictive Uncertainty Estimation via Prior Networkscited by 557
  15. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 384
  16. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
  17. 2019Deep Evidential Regressioncited by 355
  18. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 340
  19. 2018Selective Experience Replay for Lifelong Learningcited by 241
  20. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 219
  21. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learningcited by 215
  22. 2017Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translationcited by 167
  23. 2020Posterior Network: Uncertainty Estimation without OOD Samples via Density-Based Pseudo-Countscited by 164
  24. 2022TransDreamer: Reinforcement Learning with Transformer World Modelscited by 143
  25. 2022Jump-Start Reinforcement Learningcited by 112
  26. 2021Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observabilitycited by 109
  27. 2021DEUP: Direct Epistemic Uncertainty Predictioncited by 104
  28. 2018Safe Reinforcement Learning with Model Uncertainty Estimatescited by 72
  29. 2019Estimating Risk and Uncertainty in Deep Reinforcement Learningcited by 70
  30. 2020Smooth Exploration for Robotic Reinforcement Learningcited by 54
  31. 2016Model-Free Imitation Learning with Policy Optimizationcited by 52
  32. 2018Information-Directed Exploration for Deep Reinforcement Learningcited by 52
  33. 2021MobILE: Model-Based Imitation Learning From Observation Alonecited by 44
  34. 2022Disentangling Epistemic and Aleatoric Uncertainty in Reinforcement Learningcited by 27
  35. 2017Efficient exploration with Double Uncertain Value Networkscited by 16
  36. 2020Expert-Supervised Reinforcement Learning for Offline Policy Learning and Evaluationcited by 14
  37. 2018Pretraining Deep Actor-Critic Reinforcement Learning Algorithms With Expert Demonstrationscited by 7
  38. 2021Enabling risk-aware Reinforcement Learning for medical interventions through uncertainty decompositioncited by 6

50 most cited of 51 All 51 (JSON)

Built on by

Not cited yet.