mlsanityData to 7 Oct 2026

Upper Counterfactual Confidence Bounds: a New Optimism Principle for Contextual Bandits

arXiv 2007.07876 ↗First posted 15 Jul 2020cs.LG math.ST stat.MLcited by 35

Not on the map. Why?

Abstract (arXiv)

The principle of optimism in the face of uncertainty is one of the most widely used and successful ideas in multi-armed bandits and reinforcement learning. However, existing optimistic algorithms (primarily UCB and its variants) often struggle to deal with general function classes and large context spaces. In this paper, we study general contextual bandits with an offline regression oracle and propose a simple, generic principle to design optimistic algorithms, dubbed "Upper Counterfactual Confidence Bounds" (UCCB). The key innovation of UCCB is building confidence bounds in policy space, rather than in action space as is done in UCB. We demonstrate that these algorithms are provably optimal and computationally efficient in handling general function classes and large context spaces. Furthermore, we illustrate that the UCCB principle can be seamlessly extended to infinite-action general contextual bandits, provide the first solutions to these settings when employing an offline regression oracle.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 4; 2023: 6; 2024: 8; 2025: 6; 2026: 6.

35 papers.

Sort:

By area:

  1. 2021Adapting to Misspecification in Contextual Banditscited by 81
  2. 2021Efficient First-Order Contextual Bandits: Prediction, Allocation, and Triangular Discriminationcited by 53
  3. 2021Provable Model-based Nonlinear Bandit and Reinforcement Learning: Shelve Optimism, Embrace Virtual Curvaturecited by 32
  4. 2022Contextual Bandits with Large Action Spaces: Made Practicalcited by 32
  5. 2021Adapting to Misspecification in Contextual Bandits with Offline Regression Oraclescited by 21
  6. 2023Bypassing the Simulator: Near-Optimal Adversarial Linear Contextual Banditscited by 18
  7. 2022Contextual Bandits with Smooth Regret: Efficient Learning in Continuous Action Spacescited by 17
  8. 2023Bayesian Design Principles for Frequentist Sequential Learningcited by 17
  9. 2022Optimism in Face of a Context: Regret Guarantees for Stochastic Contextual MDPcited by 11
  10. 2022Eluder-based Regret for Stochastic Contextual MDPscited by 9
  11. 2023Efficient Rate Optimal Regret for Adversarial Contextual MDPs Using Online Function Approximationcited by 9
  12. 2024How Does Variance Shape the Regret in Contextual Bandits?cited by 9
23 more
  1. 2024Offline Oracle-Efficient Learning for Contextual MDPs via Layerwise Exploration-Exploitation Tradeoffcited by 7
  2. 2024Efficient Contextual Bandits with Uninformed Feedback Graphscited by 6
  3. 2024Diffusion Models Meet Contextual Banditscited by 5
  4. 2024Accelerated Preference Optimization for Large Language Model Alignmentcited by 5
  5. 2025Regret Bounds for Adversarial Contextual Bandits with General Function Approximation and Delayed Feedbackcited by 5
  6. 2023Online Learning in Contextual Second-Price Pay-Per-Click Auctionscited by 4
  7. 2024Sample Complexity Characterization for Linear Contextual MDPscited by 4
  8. 2024Contextual Multinomial Logit Bandits with General Value Functionscited by 4
  9. 2025Learning to Price with Resource Constraints: From Full Information to Machine-Learned Pricescited by 3
  10. 2023Infinite Action Contextual Bandits with Reusable Data Exhaustcited by 2
  11. 2024On the Optimal Regret of Locally Private Linear Contextual Banditcited by 2
  12. 2025Minimax Optimal Reinforcement Learning with Quasi-Optimismcited by 2
  13. 2026Taming the Monster Every Context: Complexity Measure and Unified Framework for Offline-Oracle Efficient Contextual Banditscited by 2
  14. 2023Harnessing the Power of Federated Learning in Federated Contextual Banditscited by 1
  15. 2026Optimal Regret for Policy Optimization in Contextual Banditscited by 1
  16. 2026Collaborative Contextual Bayesian Optimizationcited by 1
  17. 2021Learning Rates for Nonconvex Pairwise Learningcited by 0
  18. 2025Constrained Online Decision-Making: A Unified Frameworkcited by 0
  19. 2025Adaptive Data Augmentation for Thompson Samplingcited by 0
  20. 2025Exploration via Feature Perturbation in Contextual Banditscited by 0
  21. 2026Near-Optimal Regret for Policy Optimization in Contextual MDPs with General Offline Function Approximationcited by 0
  22. 2026Optimal and Efficient Contextual Combinatorial Semi-bandits with General Function Approximationcited by 0
  23. 2026When Greedy Sampling Explores: KL-Regularized Contextual Bandits without Eluder-Dimension Dependencecited by 0