# Safe Exploration for Optimizing Contextual Bandits

arXiv: https://arxiv.org/abs/2002.00467 · HTML: https://mlsanity.com/paper/2002.00467/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Feb 2020.

## Built on by (most cited first, 14 of 14)

- [Doubly-Robust Estimation for Correcting Position-Bias in Click Feedback for Unbiased Learning to Rank](https://mlsanity.com/paper/2203.17118/) (2022)
- [Safe Deployment for Counterfactual Learning to Rank with Exposure-Based Risk Minimization](https://mlsanity.com/paper/2305.01522/) (2023)
- [Practical Bandits: An Industry Perspective](https://mlsanity.com/paper/2302.01223/) (2023)
- [Practical and Robust Safety Guarantees for Advanced Counterfactual Learning to Rank](https://mlsanity.com/paper/2407.19943/) (2024)
- [Robust Generalization and Safe Query-Specialization in Counterfactual Learning to Rank](https://mlsanity.com/paper/2102.05990/) (2021)
- [Constrained Policy Optimization for Controlled Self-Learning in Conversational AI Systems](https://mlsanity.com/paper/2209.08429/) (2022)
- [Proximal Ranking Policy Optimization for Practical Safety in Counterfactual Learning to Rank](https://mlsanity.com/paper/2409.09881/) (2024)
- [The Minimal Search Space for Conditional Causal Bandits](https://mlsanity.com/paper/2502.06577/) (2025)
- [Constrained Online Decision-Making: A Unified Framework](https://mlsanity.com/paper/2505.07101/) (2025)
- [Contextual Bandits with Sparse Data in Web setting](https://mlsanity.com/paper/2105.02873/) (2021)
- [A Deep Learning Approach for Automatic Detection of Qualitative Features of Lecturing](https://mlsanity.com/paper/2205.14919/) (2022)
- [BALLAST: Bandit-Assisted Learning for Latency-Aware Stable Timeouts in Raft](https://mlsanity.com/paper/2512.21165/) (2025)
- [Multi-Task Representation Learning for Conservative Linear Bandits](https://mlsanity.com/paper/2605.12176/) (2026)
- [The Gold in Bias: Maturing the AI Design Process through Verification](https://mlsanity.com/paper/2609.29730/) (2026)

## Abstract (arXiv)

Contextual bandit problems are a natural fit for many information retrieval tasks, such as learning to rank, text classification, recommendation, etc. However, existing learning methods for contextual bandit problems have one of two drawbacks: they either do not explore the space of all possible document rankings (i.e., actions) and, thus, may miss the optimal ranking, or they present suboptimal rankings to a user and, thus, may harm the user experience. We introduce a new learning method for contextual bandit problems, Safe Exploration Algorithm (SEA), which overcomes the above drawbacks. SEA starts by using a baseline (or production) ranking system (i.e., policy), which does not harm the user experience and, thus, is safe to execute, but has suboptimal performance and, thus, needs to be improved. Then SEA uses counterfactual learning to learn a new policy based on the behavior of the baseline policy. SEA also uses high-confidence off-policy evaluation to estimate the performance of the newly learned policy. Once the performance of the newly learned policy is at least as good as the performance of the baseline policy, SEA starts using the new policy to execute new actions, allowing it to actively explore favorable regions of the action space. This way, SEA never performs worse than the baseline policy and, thus, does not harm the user experience, while still exploring the action space and, thus, being able to find an optimal policy. Our experiments using text classification and document retrieval confirm the above by comparing SEA (and a boundless variant called BSEA) to online and offline learning methods for contextual bandit problems.