# Warm-starting Contextual Bandits: Robustly Combining Supervised and Bandit Feedback

arXiv: https://arxiv.org/abs/1901.00301 · HTML: https://mlsanity.com/paper/1901.00301/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jan 2019.

## Built on by (most cited first, 21 of 21)

- [LLMs Are In-Context Bandit Reinforcement Learners](https://mlsanity.com/paper/2410.05362/) (2024)
- [Multi-Armed Bandits With Machine Learning-Generated Surrogate Rewards](https://mlsanity.com/paper/2506.16658/) (2025)
- [Artificial Replay: A Meta-Algorithm for Harnessing Historical Data in Bandits](https://mlsanity.com/paper/2210.00025/) (2022)
- [Leveraging Demonstrations to Improve Online Learning: Quality Matters](https://mlsanity.com/paper/2302.03319/) (2023)
- [Jump Starting Bandits with LLM-Generated Prior Knowledge](https://mlsanity.com/paper/2406.19317/) (2024)
- [Online Bandit Learning with Offline Preference Data for Improved RLHF](https://mlsanity.com/paper/2406.09574/) (2024)
- [Best Arm Identification with Possibly Biased Offline Data](https://mlsanity.com/paper/2505.23165/) (2025)
- [Contextual Online Pricing with (Biased) Offline Data](https://mlsanity.com/paper/2507.02762/) (2025)
- [Thompson Sampling for Robust Transfer in Multi-Task Bandits](https://mlsanity.com/paper/2206.08556/) (2022)
- [Augmenting Online RL with Offline Data is All You Need: A Unified Hybrid RL Algorithm Design and Analysis](https://mlsanity.com/paper/2505.13768/) (2025)
- [Interactive and Hybrid Imitation Learning: Provably Beating Behavior Cloning](https://mlsanity.com/paper/2412.07057/) (2024)
- [Direction-Aware Offline-to-Online Learning in Linear Contextual Bandits](https://mlsanity.com/paper/2604.24016/) (2026)
- [Bandits with Stochastic Experts: Constant Regret, Empirical Experts and Episodes](https://mlsanity.com/paper/2107.03263/) (2021)
- [Leveraging User-Triggered Supervision in Contextual Bandits](https://mlsanity.com/paper/2302.03784/) (2023)
- [Identifiable Latent Bandits: Leveraging observational data for personalized decision-making](https://mlsanity.com/paper/2407.16239/) (2024)
- [MBExplainer: Multilevel bandit-based explanations for downstream models with augmented graph embeddings](https://mlsanity.com/paper/2411.00287/) (2024)
- [Deconfounded Warm-Start Thompson Sampling with Applications to Precision Medicine](https://mlsanity.com/paper/2505.17283/) (2025)
- [Offline Clustering of Preference Learning with Active-data Augmentation](https://mlsanity.com/paper/2510.26301/) (2025)
- [Jump Start or False Start? A Theoretical and Empirical Evaluation of LLM-initialized Bandits](https://mlsanity.com/paper/2604.02527/) (2026)
- [Sample-Mean Anchored Thompson Sampling for Offline-to-Online Learning with Distribution Shift](https://mlsanity.com/paper/2605.10289/) (2026)
- [Personalizing Large Language Model Agents with Small Policy Models](https://mlsanity.com/paper/2608.00215/) (2026)

## Abstract (arXiv)

We investigate the feasibility of learning from a mix of both fully-labeled supervised data and contextual bandit data. We specifically consider settings in which the underlying learning signal may be different between these two data sources. Theoretically, we state and prove no-regret algorithms for learning that is robust to misaligned cost distributions between the two sources. Empirically, we evaluate some of these algorithms on a large selection of datasets, showing that our approach is both feasible and helpful in practice.