# Bayesian Q-learning With Imperfect Expert Demonstrations

arXiv: https://arxiv.org/abs/2210.01800 · HTML: https://mlsanity.com/paper/2210.01800/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 29 references that are in the map.
- First posted on arXiv 1 Oct 2022.

## Builds on (most cited first, 14 of 14)

- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [Unifying Count-Based Exploration and Intrinsic Motivation](https://mlsanity.com/paper/1606.01868/) (2016)
- [Is Q-learning Provably Efficient?](https://mlsanity.com/paper/1807.03765/) (2018)
- [Overcoming Exploration in Reinforcement Learning with Demonstrations](https://mlsanity.com/paper/1709.10089/) (2017)
- [Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations](https://mlsanity.com/paper/1904.06387/) (2019)
- [Randomized Prior Functions for Deep Reinforcement Learning](https://mlsanity.com/paper/1806.03335/) (2018)
- [Deep Exploration via Randomized Value Functions](https://mlsanity.com/paper/1703.07608/) (2017)
- [Temporally-Extended ε-Greedy Exploration](https://mlsanity.com/paper/2006.01782/) (2020)
- [Reinforcement Learning from Imperfect Demonstrations under Soft Expert Guidance](https://mlsanity.com/paper/1911.07109/) (2019)
- [Shaping Rewards for Reinforcement Learning with Imperfect Demonstrations using Generative Models](https://mlsanity.com/paper/2011.01298/) (2020)

## Built on by (most cited first, 3 of 3)

- [Correcting discount-factor mismatch in on-policy policy gradient methods](https://mlsanity.com/paper/2306.13284/) (2023)
- [Uncertainty-Based Smooth Policy Regularisation for Reinforcement Learning with Few Demonstrations](https://mlsanity.com/paper/2509.15981/) (2025)
- [Optimizing Algorithms for Mobile Health Interventions with Active Querying Optimization](https://mlsanity.com/paper/2512.08950/) (2025)

## Abstract (arXiv)

Guided exploration with expert demonstrations improves data efficiency for reinforcement learning, but current algorithms often overuse expert information. We propose a novel algorithm to speed up Q-learning with the help of a limited amount of imperfect expert demonstrations. The algorithm avoids excessive reliance on expert data by relaxing the optimal expert assumption and gradually reducing the usage of uninformative expert data. Experimentally, we evaluate our approach on a sparse-reward chain environment and six more complicated Atari games with delayed rewards. With the proposed methods, we can achieve better results than Deep Q-learning from Demonstrations (Hester et al., 2017) in most environments.