# Learning with Incomplete Context: Linear Contextual Bandits with Pretrained Imputation

arXiv: https://arxiv.org/abs/2510.09908 · HTML: https://mlsanity.com/paper/2510.09908/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 56 references that are in the map.
- First posted on arXiv 10 Oct 2025.

## Builds on (most cited first, 20 of 20)

- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [Offline Reinforcement Learning as One Big Sequence Modeling Problem](https://mlsanity.com/paper/2106.02039/) (2021)
- [Prediction-Powered Inference](https://mlsanity.com/paper/2301.09633/) (2023)
- [Supervised Pretraining Can Learn In-Context Reinforcement Learning](https://mlsanity.com/paper/2306.14892/) (2023)
- [Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretraining](https://mlsanity.com/paper/2310.08566/) (2023)
- [Personalized HeartSteps: A Reinforcement Learning Algorithm for Optimizing Physical Activity](https://mlsanity.com/paper/1909.03539/) (2019)
- [LOLA: LLM-Assisted Online Learning Algorithm for Content Experiments](https://mlsanity.com/paper/2406.02611/) (2024)
- [Contextual Linear Bandits under Noisy Features: Towards Bayesian Oracles](https://mlsanity.com/paper/1703.01347/) (2017)
- [Regret Lower Bound and Optimal Algorithm for High-Dimensional Contextual Linear Bandit](https://mlsanity.com/paper/2109.11612/) (2021)
- [Replicable Bandits for Digital Health Interventions](https://mlsanity.com/paper/2407.15377/) (2024)
- [Contextual Thompson Sampling via Generation of Missing Data](https://mlsanity.com/paper/2502.07064/) (2025)
- [Thompson Sampling in Partially Observable Contextual Bandits](https://mlsanity.com/paper/2402.10289/) (2024)
- [Partially Observable Contextual Bandits with Linear Payoffs](https://mlsanity.com/paper/2409.11521/) (2024)
- [Linear Bandits with Partially Observable Features](https://mlsanity.com/paper/2502.06142/) (2025)
- [Statistical Inference for Misspecified Contextual Bandits](https://mlsanity.com/paper/2509.06287/) (2025)
- [Online learning with Corrupted context: Corrupted Contextual Bandits](https://mlsanity.com/paper/2006.15194/) (2020)
- [Leveraging Offline Data in Linear Latent Contextual Bandits](https://mlsanity.com/paper/2405.17324/) (2024)
- [Pre-Trained AI Model Assisted Online Decision-Making under Missing Covariates: A Theoretical Perspective](https://mlsanity.com/paper/2507.07852/) (2025)
- [High dimensional stochastic linear contextual bandit with missing covariates](https://mlsanity.com/paper/2207.11165/) (2022)
- [HR-Bandit: Human-AI Collaborated Linear Recourse Bandit](https://mlsanity.com/paper/2410.14640/) (2024)

## Abstract (arXiv)

The rise of large-scale pretrained models has made it feasible to generate predictive or synthetic features at low cost, raising the question of how to incorporate such surrogate predictions into downstream decision-making. We study this problem in the setting of online linear contextual bandits, where contexts may be complex, nonstationary, and only partially observed. In addition to bandit data, we assume access to an auxiliary dataset containing fully observed contexts--common in practice since such data are collected without adaptive interventions. We propose PULSE-UCB, an algorithm that leverages pretrained models trained on the auxiliary data to impute missing features during online decision-making. We establish regret guarantees that decompose into a standard bandit term plus an additional component reflecting pretrained model quality. In the i.i.d. context case with H"older-smooth missing features, PULSE-UCB achieves near-optimal performance, supported by matching lower bounds. Our results quantify how uncertainty in predicted contexts affects decision quality and how much historical data is needed to improve downstream learning.