Flexible and Efficient Contextual Bandits with Heterogeneous Treatment Effect Oracles
On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF
Abstract (arXiv)
Contextual bandit algorithms often estimate reward models to inform decision-making. However, true rewards can contain action-independent redundancies that are not relevant for decision-making. We show it is more data-efficient to estimate any function that explains the reward differences between actions, that is, the treatment effects. Motivated by this observation, building on recent work on oracle-based bandit algorithms, we provide the first reduction of contextual bandits to general-purpose heterogeneous treatment effect estimation, and we design a simple and computationally efficient algorithm based on this reduction. Our theoretical and experimental results demonstrate that heterogeneous treatment effect estimation in contextual bandits offers practical advantages over reward estimation, including more efficient model estimation and greater flexibility to model misspecification.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 23 of its 52 references that are in the map. First posted on arXiv 30 Mar 2022.
Builds on
23 of 52 references.
References · 23
- 2017Meta-learners for Estimating Heterogeneous Treatment Effects using Machine Learningcited by 475
- 2017Quasi-Oracle Estimation of Heterogeneous Treatment Effectscited by 328
- 2016Generalized Random Forestscited by 309
- 2014Taming the Monster: A Fast and Simple Algorithm for Contextual Banditscited by 200
- 2020Beyond UCB: Optimal and Efficient Contextual Bandits with Regression Oraclescited by 195
- 2019Orthogonal Statistical Learningcited by 150
- 2019Learning with Good Feature Representations in Bandits and in RL with a Generative Modelcited by 124
- 2020Bypassing the Monster: A Faster and Simpler Optimal Algorithm for Contextual Bandits under Realizabilitycited by 114
- 2015Contextual Dueling Banditscited by 101
- 2018Practical Contextual Bandits with Regression Oraclescited by 92
- 2021Adapting to Misspecification in Contextual Banditscited by 81
- 2018A Contextual Bandit Bake-offcited by 64
11 more
- 2019Model selection for contextual banditscited by 60
- 2019Nearly Minimax-Optimal Regret for Linearly Parameterized Banditscited by 54
- 2021Efficient and Optimal Algorithms for Contextual Dueling Bandits under Realizabilitycited by 44
- 2017Misspecified Linear Banditscited by 41
- 2020Problem-Complexity Adaptive Model Selection for Stochastic Linear Banditscited by 26
- 2019Smooth Contextual Bandits: Bridging the Parametric and Non-differentiable Regret Regimescited by 25
- 2017Action Centered Contextual Banditscited by 23
- 2021Adapting to Misspecification in Contextual Bandits with Offline Regression Oraclescited by 21
- 2018Semiparametric Contextual Banditscited by 19
- 2019Smoothness-Adaptive Contextual Banditscited by 15
- 2020Towards Optimal Problem Dependent Generalization Error Bounds in Statistical Learning Theorycited by 13
Built on by
Citing papers per year: 2022: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2024Diffusion Models Meet Contextual Bandits
- 2026Gated Decoupled Compositional Bandits: A Unified Theory of Contextual Bandits with Supervised-Calibrated Action Scaling and Pre-Execution Gating
- 2023Reward Learning as Doubly Nonparametric Bandits: Optimal Design and Scaling Laws
- 2023Oracle-Efficient Pessimism: Offline Policy Optimization in Contextual Bandits
- 2023RoME: A Robust Mixed-Effects Bandit Algorithm for Optimizing Mobile Health Interventions
- 2026RIE-Greedy: Regularization-Induced Exploration for Contextual Bandits
- 2020Statistical Inference for Online Decision-Making: In a Contextual Bandit Setting
- 2024Linear Contextual Bandits with Interference