# Online learning in bandits with predicted context

arXiv: https://arxiv.org/abs/2307.13916 · HTML: https://mlsanity.com/paper/2307.13916/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 74 references that are in the map.
- First posted on arXiv 26 Jul 2023.

## Builds on (most cited first, 25 of 25)

- [A Tutorial on Thompson Sampling](https://mlsanity.com/paper/1707.02038/) (2017)
- [Optimal and Adaptive Off-policy Evaluation in Contextual Bandits](https://mlsanity.com/paper/1612.01205/) (2016)
- [Personalized HeartSteps: A Reinforcement Learning Algorithm for Optimizing Physical Activity](https://mlsanity.com/paper/1909.03539/) (2019)
- [Latent Bandits Revisited](https://mlsanity.com/paper/2006.08714/) (2020)
- [Statistical Inference with M-Estimators on Adaptively Collected Data](https://mlsanity.com/paper/2104.14074/) (2021)
- [Off-Policy Evaluation via Adaptive Weighting with Data from Contextual Bandits](https://mlsanity.com/paper/2106.02029/) (2021)
- [A Reinforcement Learning System to Encourage Physical Activity in Diabetes Patients](https://mlsanity.com/paper/1605.04070/) (2016)
- [Robust Stochastic Linear Contextual Bandits Under Adversarial Attacks](https://mlsanity.com/paper/2106.02978/) (2021)
- [Post-Contextual-Bandit Inference](https://mlsanity.com/paper/2106.00418/) (2021)
- [Power Constrained Bandits](https://mlsanity.com/paper/2004.06230/) (2020)
- [Contextual Bandits with Latent Confounders: An NMF Approach](https://mlsanity.com/paper/1606.00119/) (2016)
- [Targeting for long-term outcomes](https://mlsanity.com/paper/2010.15835/) (2020)
- [Towards Agnostic Feature-based Dynamic Pricing: Linear Policies vs Linear Valuation with Unknown Noise](https://mlsanity.com/paper/2201.11341/) (2022)
- [Policy Learning with Observational Data](https://mlsanity.com/paper/1702.02896/) (2017)
- [Latent Contextual Bandits and their Application to Personalized Recommendations for New Users](https://mlsanity.com/paper/1604.06743/) (2016)
- [Stochastic Bandits with Context Distributions](https://mlsanity.com/paper/1906.02685/) (2019)
- [Reward Design For An Online Reinforcement Learning Algorithm Supporting Oral Self-Care](https://mlsanity.com/paper/2208.07406/) (2022)
- [Non-Stationary Latent Bandits](https://mlsanity.com/paper/2012.00386/) (2020)
- [Robust Bandit Learning with Imperfect Context](https://mlsanity.com/paper/2102.05018/) (2021)
- [Stochastic Conservative Contextual Linear Bandits](https://mlsanity.com/paper/2203.15629/) (2022)
- [Online learning with Corrupted context: Corrupted Contextual Bandits](https://mlsanity.com/paper/2006.15194/) (2020)
- [A New Bandit Setting Balancing Information from State Evolution and Corrupted Context](https://mlsanity.com/paper/2011.07989/) (2020)
- [Worst-case Performance of Greedy Policies in Bandits with Imperfect Context Observations](https://mlsanity.com/paper/2204.04773/) (2022)
- [Information-Gathering in Latent Bandits](https://mlsanity.com/paper/2207.03635/) (2022)
- [Sample Size Calculations for Micro-randomized Trials in mHealth](https://mlsanity.com/paper/1504.00238/) (2015)

## Built on by (most cited first, 4 of 4)

- [Partially Observable Contextual Bandits with Linear Payoffs](https://mlsanity.com/paper/2409.11521/) (2024)
- [Reinforcement Learning in the Real World: A Survey of Statistical Challenges and Future Directions](https://mlsanity.com/paper/2601.15353/) (2026)
- [Thompson Sampling for Stochastic Bandits with Noisy Contexts: An Information-Theoretic Regret Analysis](https://mlsanity.com/paper/2401.11565/) (2024)
- [The Fallacy of Minimizing Cumulative Regret in the Sequential Task Setting](https://mlsanity.com/paper/2403.10946/) (2024)

## Abstract (arXiv)

We consider the contextual bandit problem where at each time, the agent only has access to a noisy version of the context and the error variance (or an estimator of this variance). This setting is motivated by a wide range of applications where the true context for decision-making is unobserved, and only a prediction of the context by a potentially complex machine learning algorithm is available. When the context error is non-vanishing, classical bandit algorithms fail to achieve sublinear regret. We propose the first online algorithm in this setting with sublinear regret guarantees under mild conditions. The key idea is to extend the measurement error model in classical statistics to the online decision-making setting, which is nontrivial due to the policy being dependent on the noisy context observations. We further demonstrate the benefits of the proposed approach in simulation environments based on synthetic and real digital intervention datasets.