mlsanityData to 7 Oct 2026

Online Statistical Inference for Contextual Bandits via Stochastic Gradient Descent

arXiv 2212.14883 ↗First posted 30 Dec 2022stat.ML cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

With the fast development of big data, learning the optimal decision rule by recursively updating it and making online decisions has been easier than before. We study the online statistical inference of model parameters in a contextual bandit framework of sequential decision-making. We propose a general framework for an online and adaptive data collection environment that can update decision rules via weighted stochastic gradient descent. We allow different weighting schemes of the stochastic gradient and establish the asymptotic normality of the parameter estimator. Our proposed estimator significantly improves the asymptotic efficiency over the previous averaged SGD approach via inverse probability weights. We also conduct an optimality analysis on the weights in a linear regression setting. We provide a Bahadur representation of the proposed estimator and show that the remainder term in the Bahadur representation entails a slower convergence rate compared to classical SGD due to the adaptive data collection.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 18 of its 45 references that are in the map. First posted on arXiv 30 Dec 2022.

Builds on

18 of 45 references.

References · 18

  1. 2019Confidence Intervals for Policy Evaluation in Adaptive Experimentscited by 95
  2. 2016Statistical Inference for Model Parameters in Stochastic Gradient Descentcited by 71
  3. 2020Statistical Inference of the Value Function for Reinforcement Learning in Infinite Horizon Settingscited by 65
  4. 2021Statistical Inference with M-Estimators on Adaptively Collected Datacited by 40
  5. 2021Off-Policy Evaluation via Adaptive Weighting with Data from Contextual Banditscited by 40
  6. 2017Accurate Inference for Adaptive Linear Modelscited by 39
  7. 2016Asymptotic Optimality in Stochastic Optimizationcited by 38
  8. 2020Online Covariance Matrix Estimation in Stochastic Gradient Descentcited by 37
  9. 2019Bootstrapping Upper Confidence Boundcited by 36
  10. 2022Off-Policy Confidence Interval Estimation with Confounded Markov Decision Processcited by 31
  11. 2016Dynamic Pricing with Demand Covariatescited by 30
  12. 2021Fast and Robust Online Inference with Stochastic Gradient Descent via Random Scalingcited by 28
6 more
  1. 2021Online Bootstrap Inference For Policy Evaluation in Reinforcement Learningcited by 26
  2. 2021Near-optimal inference in adaptive linear regressioncited by 20
  3. 2020Statistical Inference for Online Decision-Making: In a Contextual Bandit Settingcited by 17
  4. 2020Statistical Inference for Online Decision Making via Stochastic Gradient Descentcited by 17
  5. 2021Online Statistical Inference for Stochastic Optimization via Kiefer-Wolfowitz Methodscited by 14
  6. 2022Reinforcement Learning with Heterogeneous Data: Estimation and Inferencecited by 7

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 4; 2025: 2.

7 papers.

  1. 2025Gaussian Approximation and Multiplier Bootstrap for Stochastic Gradient Descentcited by 8
  2. 2023Optimal Multitask Linear Regression and Contextual Bandits under Sparse Heterogeneitycited by 7
  3. 2024Allocation Stability and Wald Inference under Variance-Aware UCBcited by 4
  4. 2024Online Policy Learning and Inference by Matrix Completioncited by 3
  5. 2024Asymptotic Time-Uniform Inference for Parameters in Averaged Stochastic Approximationcited by 3
  6. 2024Asymptotic Analysis of Sample-averaged Q-learningcited by 2
  7. 2025Statistical Inference for Differentially Private Stochastic Gradient Descentcited by 2