mlsanityData to 7 Oct 2026

Leveraging Demonstrations to Improve Online Learning: Quality Matters

arXiv 2302.03319 ↗First posted 7 Feb 2023cs.LG math.ST stat.MLcited by 13

Not on the map. Why?

Abstract (arXiv)

We investigate the extent to which offline demonstration data can improve online learning. It is natural to expect some improvement, but the question is how, and by how much? We show that the degree of improvement must depend on the quality of the demonstration data. To generate portable insights, we focus on Thompson sampling (TS) applied to a multi-armed bandit as a prototypical online learning algorithm and model. The demonstration data is generated by an expert with a given competence level, a notion we introduce. We propose an informed TS algorithm that utilizes the demonstration data in a coherent way through Bayes' rule and derive a prior-dependent Bayesian regret bound. This offers insight into how pretraining can greatly improve online performance and how the degree of improvement increases with the expert's competence level. We also develop a practical, approximate informed TS algorithm through Bayesian bootstrapping and show substantial empirical regret reduction through experiments.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 30 of its 36 references that are in the map. First posted on arXiv 7 Feb 2023.

Builds on

30 of 36 references.

References · 30

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  3. 2016CVXPY: A Python-Embedded Modeling Language for Convex Optimizationcited by 633
  4. 2017A Tutorial on Thompson Samplingcited by 420
  5. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 275
  6. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 218
  7. 2014An Information-Theoretic Analysis of Thompson Samplingcited by 178
  8. 2014Learning to Optimize via Information-Directed Samplingcited by 168
  9. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
  10. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learningcited by 157
  11. 2017Deep Exploration via Randomized Value Functionscited by 155
  12. 2021Epistemic Neural Networkscited by 120
18 more
  1. 2021Feel-Good Thompson Sampling for Contextual Bandits and Reinforcement Learningcited by 65
  2. 2017Ensemble Samplingcited by 62
  3. 2021Meta-Thompson Samplingcited by 62
  4. 2022Imitation Learning by Estimating Expertise of Demonstratorscited by 50
  5. 2021Bayesian decision-making under misspecified priors with applications to meta-learningcited by 43
  6. 2022Leveraging Offline Data in Online Reinforcement Learningcited by 42
  7. 2019An Information-Theoretic Approach to Minimax Regret in Partial Monitoringcited by 41
  8. 2013Prior-free and prior-dependent regret bounds for Thompson Samplingcited by 25
  9. 2022Regret Bounds for Information-Directed Reinforcement Learningcited by 24
  10. 2022An Analysis of Ensemble Samplingcited by 23
  11. 2021The Neural Testbed: Evaluating Joint Predictionscited by 22
  12. 2019Warm-starting Contextual Bandits: Robustly Combining Supervised and Bandit Feedbackcited by 21
  13. 2021Information Directed Sampling for Sparse Linear Banditscited by 20
  14. 2022Ensembles for Uncertainty Estimation: Benefits of Prior Functions and Bootstrappingcited by 19
  15. 2022Artificial Replay: A Meta-Algorithm for Harnessing Historical Data in Banditscited by 12
  16. 2017Thompson Sampling For Stochastic Bandits with Graph Feedbackcited by 9
  17. 2022Leveraging Initial Hints for Free in Stochastic Linear Banditscited by 7
  18. 2018Bandits with Side Observations: Bounded vs. Logarithmic Regretcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 3; 2025: 3; 2026: 5.

13 papers.

Sort:

By area:

  1. 2025Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuningcited by 13
  2. 2024Online Bandit Learning with Offline Preference Data for Improved RLHFcited by 6
  3. 2025Best Arm Identification with Possibly Biased Offline Datacited by 4
  4. 2026Reinforcement Learning from Human Feedback: A Statistical Perspectivecited by 4
  5. 2023Efficient Online Learning with Offline Datasets for Infinite Horizon MDPs: A Bayesian Approachcited by 3
  6. 2024Constrained Online Two-stage Stochastic Optimization: Algorithm with (and without) Predictionscited by 1
  7. 2024Sequential Decision Making with Expert Demonstrations under Unobserved Heterogeneitycited by 1
  8. 2023Sequential Best-Arm Identification with Application to Brain-Computer Interfacecited by 0
  9. 2025Deconfounded Warm-Start Thompson Sampling with Applications to Precision Medicinecited by 0
  10. 2026Offline-to-Online Learning in Linear Banditscited by 0
  11. 2026Expert Behavior Prior Reinforcement Learningcited by 0
  12. 2026VERAGMIL: Virtual Environment for Scooping Granular Foods with Imitation Learning Modelscited by 0
1 more
  1. 2026Improving Online Reinforcement Learning via Bidirectional Behavior Prior Distillationcited by 0