mlsanityData to 7 Oct 2026

A Contextual Bandit Bake-off

arXiv 1802.04064 ↗First posted 12 Feb 2018stat.ML cs.LGcited by 64

Not on the map. Why?

Abstract (arXiv)

Contextual bandit algorithms are essential for solving many real-world interactive machine learning problems. Despite multiple recent successes on statistically and computationally efficient methods, the practical behavior of these algorithms is still poorly understood. We leverage the availability of large numbers of supervised learning datasets to empirically evaluate contextual bandit algorithms, focusing on practical methods that learn by relying on optimization oracles from supervised learning. We find that a recent method (Foster et al., 2018) using optimism under uncertainty works the best overall. A surprisingly close second is a simple greedy baseline that only explores implicitly through the diversity of contexts, followed by a variant of Online Cover (Agarwal et al., 2014) which tends to be more conservative but robust to problem specification by design. Along the way, we also evaluate various components of contextual bandit algorithm design such as loss estimators. Overall, this is a thorough study and review of contextual bandit methodology.

Key facts. Cited by 64 papers in the map (data through 7 Oct 2026). First posted on arXiv 12 Feb 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 15; 2023: 11; 2024: 9; 2025: 6; 2026: 11.

64 papers.

Sort:

By area:

  1. 2021Efficient First-Order Contextual Bandits: Prediction, Allocation, and Triangular Discriminationcited by 53
  2. 2022Contextual Bandits with Large Action Spaces: Made Practicalcited by 32
  3. 2022Contextual Bandits with Packing and Covering Constraints: A Modular Lagrangian Approach via Regressioncited by 30
  4. 2021Generalized Linear Bandits with Local Differential Privacycited by 29
  5. 2024LLMs Are In-Context Bandit Reinforcement Learnerscited by 27
  6. 2022Conformal Off-Policy Prediction in Contextual Banditscited by 25
  7. 2021Online Multi-Armed Bandits with Adaptive Inferencecited by 21
  8. 2021Invariant Policy Learning: A Causal Perspectivecited by 17
  9. 2021Patterns, predictions, and actions: A story about machine learningcited by 14
  10. 2021Learning Accurate Decision Trees with Bandit Feedback via Quantized Gradient Descentcited by 14
  11. 2021On component interactions in two-stage recommender systemscited by 14
  12. 2022Simulating Bandit Learning from User Feedback for Extractive Question Answeringcited by 14
38 more
  1. 2023Oracle-Efficient Pessimism: Offline Policy Optimization in Contextual Banditscited by 14
  2. 2022Optimal Contextual Bandits with Knapsacks under Realizability via Regression Oraclescited by 12
  3. 2022Autothrottle: A Practical Bi-Level Approach to Resource Management for SLO-Targeted Microservicescited by 12
  4. 2023Continually Improving Extractive QA via Human Feedbackcited by 11
  5. 2022Tsetlin Machine for Solving Contextual Bandit Problemscited by 10
  6. 2023Strategic Apple Tastingcited by 10
  7. 2024Dynamic Reward Adjustment in Multi-Reward Reinforcement Learning for Counselor Reflection Generationcited by 10
  8. 2022Personalized Reward Learning with Interaction-Grounded Learning (IGL)cited by 9
  9. 2023Pearl: A Production-ready Reinforcement Learning Agentcited by 9
  10. 2021Top-k eXtreme Contextual Bandits with Arm Hierarchycited by 8
  11. 2021Be Greedy in Multi-Armed Banditscited by 6
  12. 2022Variance-Optimal Augmentation Logging for Counterfactual Evaluation in Contextual Banditscited by 6
  13. 2022Contextual Bandits in a Survey Experiment on Charitable Giving: Within-Experiment Outcomes versus Policy Learningcited by 6
  14. 2024No-Regret is not enough! Bandits with General Constraints through Adaptive Regret Minimizationcited by 6
  15. 2022Integrating Reward Maximization and Population Estimation: Sequential Decision-Making for Internal Revenue Service Audit Selectioncited by 5
  16. 2024Thompson Sampling in Partially Observable Contextual Banditscited by 5
  17. 2021Parallelizing Contextual Banditscited by 4
  18. 2025Feel-Good Thompson Sampling for Contextual Bandits: a Markov Chain Monte Carlo Showdowncited by 4
  19. 2021Optimal Regret Algorithm for Pseudo-1d Bandit Convex Optimizationcited by 3
  20. 2022RISE: Robust Individualized Decision Learning with Sensitive Variablescited by 3
  21. 2023Multiplier Bootstrap-based Explorationcited by 3
  22. 2023Geometry-Aware Approaches for Balancing Performance and Theoretical Guarantees in Linear Banditscited by 3
  23. 2024On Bits and Bandits: Quantifying the Regret-Information Trade-offcited by 3
  24. 2024Robust Offline Policy Learning with Observational Data from Multiple Sourcescited by 3
  25. 2023Infinite Action Contextual Bandits with Reusable Data Exhaustcited by 2
  26. 2023An Empirical Evaluation of Federated Contextual Bandit Algorithmscited by 2
  27. 2026Taming the Monster Every Context: Complexity Measure and Unified Framework for Offline-Oracle Efficient Contextual Banditscited by 2
  28. 2021An Efficient Algorithm for Deep Stochastic Contextual Banditscited by 1
  29. 2022Flexible and Efficient Contextual Bandits with Heterogeneous Treatment Effect Oraclescited by 1
  30. 2022Maximum entropy exploration in contextual bandits with neural networks and energy based modelscited by 1
  31. 2024A Bag of Tricks for Scaling CPU-based Deep FFMs to more than 300m Predictions per Secondcited by 1
  32. 2025Greedy Algorithm for Structured Bandits: A Sharp Characterization of Asymptotic Success / Failurecited by 1
  33. 2025Multi-Objective Reinforcement Learning for Large Language Model Optimization: Visionary Perspectivecited by 1
  34. 2026Optimal Regret for Policy Optimization in Contextual Banditscited by 1
  35. 2026Learning to Orchestrate Agents under Uncertaintycited by 1
  36. 2026Position: RL Researchers Need to Distinguish Between Solving Simulators and Using Simulators as a Proxycited by 1
  37. 2026What Does an LLM-Agent Leaderboard Rank Actually Compare?cited by 1
  38. 2022Eigen Memory Treescited by 0

50 most cited of 64 All 64 (JSON)