mlsanityData to 14 Oct 2026

Online learning in bandits with predicted context

arXiv 2307.13916 ↗First posted 26 Jul 2023stat.ML cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

We consider the contextual bandit problem where at each time, the agent only has access to a noisy version of the context and the error variance (or an estimator of this variance). This setting is motivated by a wide range of applications where the true context for decision-making is unobserved, and only a prediction of the context by a potentially complex machine learning algorithm is available. When the context error is non-vanishing, classical bandit algorithms fail to achieve sublinear regret. We propose the first online algorithm in this setting with sublinear regret guarantees under mild conditions. The key idea is to extend the measurement error model in classical statistics to the online decision-making setting, which is nontrivial due to the policy being dependent on the noisy context observations. We further demonstrate the benefits of the proposed approach in simulation environments based on synthetic and real digital intervention datasets.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 25 of its 74 references that are in the map. First posted on arXiv 26 Jul 2023.

Builds on

25 of 74 references.

References · 25

  1. 2017A Tutorial on Thompson Samplingcited by 433
  2. 2016Optimal and Adaptive Off-policy Evaluation in Contextual Banditscited by 111
  3. 2019Personalized HeartSteps: A Reinforcement Learning Algorithm for Optimizing Physical Activitycited by 70
  4. 2020Latent Bandits Revisitedcited by 44
  5. 2021Statistical Inference with M-Estimators on Adaptively Collected Datacited by 40
  6. 2021Off-Policy Evaluation via Adaptive Weighting with Data from Contextual Banditscited by 40
  7. 2016A Reinforcement Learning System to Encourage Physical Activity in Diabetes Patientscited by 38
  8. 2021Robust Stochastic Linear Contextual Bandits Under Adversarial Attackscited by 33
  9. 2021Post-Contextual-Bandit Inferencecited by 32
  10. 2020Power Constrained Banditscited by 30
  11. 2016Contextual Bandits with Latent Confounders: An NMF Approachcited by 26
  12. 2020Targeting for long-term outcomescited by 23
13 more
  1. 2022Towards Agnostic Feature-based Dynamic Pricing: Linear Policies vs Linear Valuation with Unknown Noisecited by 21
  2. 2017Policy Learning with Observational Datacited by 19
  3. 2016Latent Contextual Bandits and their Application to Personalized Recommendations for New Userscited by 18
  4. 2019Stochastic Bandits with Context Distributionscited by 18
  5. 2022Reward Design For An Online Reinforcement Learning Algorithm Supporting Oral Self-Carecited by 18
  6. 2020Non-Stationary Latent Banditscited by 12
  7. 2021Robust Bandit Learning with Imperfect Contextcited by 8
  8. 2022Stochastic Conservative Contextual Linear Banditscited by 6
  9. 2020Online learning with Corrupted context: Corrupted Contextual Banditscited by 3
  10. 2020A New Bandit Setting Balancing Information from State Evolution and Corrupted Contextcited by 3
  11. 2022Worst-case Performance of Greedy Policies in Bandits with Imperfect Context Observationscited by 2
  12. 2022Information-Gathering in Latent Banditscited by 2
  13. 2015Sample Size Calculations for Micro-randomized Trials in mHealthcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 0; 2026: 1.

4 papers.

  1. 2024Partially Observable Contextual Bandits with Linear Payoffscited by 5
  2. 2026Reinforcement Learning in the Real World: A Survey of Statistical Challenges and Future Directionscited by 5
  3. 2024Thompson Sampling for Stochastic Bandits with Noisy Contexts: An Information-Theoretic Regret Analysiscited by 2
  4. 2024The Fallacy of Minimizing Cumulative Regret in the Sequential Task Settingcited by 1