# Safe Data Collection for Offline and Online Policy Learning

arXiv: https://arxiv.org/abs/2111.04835 · HTML: https://mlsanity.com/paper/2111.04835/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 62 references that are in the map.
- First posted on arXiv 8 Nov 2021.

## Builds on (most cited first, 19 of 19)

- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [Introduction to Multi-Armed Bandits](https://mlsanity.com/paper/1904.07272/) (2019)
- [Online Learning: A Modern Introduction Using Convex Optimization](https://mlsanity.com/paper/1912.13213/) (2019)
- [Doubly Robust Policy Evaluation and Optimization](https://mlsanity.com/paper/1503.02834/) (2015)
- [Safe Policy Improvement with Baseline Bootstrapping](https://mlsanity.com/paper/1712.06924/) (2017)
- [Optimal and Adaptive Off-policy Evaluation in Contextual Bandits](https://mlsanity.com/paper/1612.01205/) (2016)
- [Off-policy evaluation for slate recommendation](https://mlsanity.com/paper/1605.04812/) (2016)
- [Batched Multi-armed Bandits Problem](https://mlsanity.com/paper/1904.01763/) (2019)
- [Cascading Bandits: Learning to Rank in the Cascade Model](https://mlsanity.com/paper/1502.02763/) (2015)
- [Inference for Batched Bandits](https://mlsanity.com/paper/2002.03217/) (2020)
- [Meta-Thompson Sampling](https://mlsanity.com/paper/2102.06129/) (2021)
- [Conservative Bandits](https://mlsanity.com/paper/1602.04282/) (2016)
- [Conservative Contextual Linear Bandits](https://mlsanity.com/paper/1611.06426/) (2016)
- [Bayesian decision-making under misspecified priors with applications to meta-learning](https://mlsanity.com/paper/2107.01509/) (2021)
- [Meta Dynamic Pricing: Transfer Learning Across Experiments](https://mlsanity.com/paper/1902.10918/) (2019)
- [Why Adaptively Collected Data Have Negative Bias and How to Correct for It](https://mlsanity.com/paper/1708.01977/) (2017)
- [Are sample means in multi-armed bandits positively or negatively biased?](https://mlsanity.com/paper/1905.11397/) (2019)
- [Control Variates for Slate Off-Policy Evaluation](https://mlsanity.com/paper/2106.07914/) (2021)
- [Phase Transitions in Bandits with Switching Constraints](https://mlsanity.com/paper/1905.10825/) (2019)

## Built on by (most cited first, 4 of 4)

- [SPEED: Experimental Design for Policy Evaluation in Linear Heteroscedastic Bandits](https://mlsanity.com/paper/2301.12357/) (2023)
- [SaVeR: Optimal Data Collection Strategy for Safe Policy Evaluation in Tabular MDP](https://mlsanity.com/paper/2406.02165/) (2024)
- [Risk-Aware Linear Bandits: Theory and Applications in Smart Order Routing](https://mlsanity.com/paper/2208.02389/) (2022)
- [Efficient and Interpretable Bandit Algorithms](https://mlsanity.com/paper/2310.14751/) (2023)

## Abstract (arXiv)

Motivated by practical needs of experimentation and policy learning in online platforms, we study the problem of safe data collection. Specifically, our goal is to develop a logging policy that efficiently explores different actions to elicit information while achieving competitive reward with a baseline production policy. We first show that a common practice of mixing the production policy with randomized exploration, despite being safe, is sub-optimal in maximizing information gain. Then, we propose a safe optimal logging policy via a novel water-filling technique for the case when no side information about the actions' expected reward is available. We improve upon this design by considering side information and also extend our approaches to the linear contextual model to account for a large number of actions. Along the way, we analyze how our data logging policies impact errors in off(line)-policy learning and empirically validate the benefit of our design by conducting extensive numerical experiments with synthetic and MNIST datasets. To further demonstrate the generality of our approach, we also consider the safe online learning setting. By adaptively applying our techniques, we develop the Safe Phased-Elimination (SafePE) algorithm that can achieve optimal regret bound with only logarithmic number of policy updates.