mlsanityData to 7 Oct 2026

Variance-Aware Linear UCB with Deep Representation for Neural Contextual Bandits

arXiv 2411.05979 ↗First posted 8 Nov 2024cs.LG stat.MLcited by 3

Not on the map. Why?

Abstract (arXiv)

By leveraging the representation power of deep neural networks, neural upper confidence bound (UCB) algorithms have shown success in contextual bandits. To further balance the exploration and exploitation, we propose Neural-σ2-LinearUCB, a variance-aware algorithm that utilizes σ2t, i.e., an upper bound of the reward noise variance at round t, to enhance the uncertainty quantification quality of the UCB, resulting in a regret performance improvement. We provide an oracle version for our algorithm characterized by an oracle variance upper bound σ2t and a practical version with a novel estimation for this variance bound. Theoretically, we provide rigorous regret analysis for both versions and prove that our oracle algorithm achieves a better regret guarantee than other neural-UCB algorithms in the neural contextual bandits setting. Empirically, our practical method enjoys a similar computational efficiency, while outperforming state-of-the-art techniques by having a better calibration and lower regret across multiple standard settings, including on the synthetic, UCI, MNIST, and CIFAR-10 datasets.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 34 of its 69 references that are in the map. First posted on arXiv 8 Nov 2024.

Builds on

34 of 69 references.

References · 34

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  3. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  4. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  5. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  6. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  7. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  8. 2018Accurate Uncertainties for Deep Learning Using Calibrated Regressioncited by 417
  9. 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 221
  10. 2014Taming the Monster: A Fast and Simple Algorithm for Contextual Banditscited by 200
  11. 2020Nearly Minimax Optimal Reinforcement Learning for Linear Mixture Markov Decision Processescited by 199
  12. 2020Beyond UCB: Optimal and Efficient Contextual Bandits with Regression Oraclescited by 195
22 more
  1. 2013Finite-Time Analysis of Kernelised Contextual Banditscited by 177
  2. 2018Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Samplingcited by 166
  3. 2014Statistical guarantees for the EM algorithm: From population to sample-based analysiscited by 131
  4. 2020Neural Thompson Samplingcited by 127
  5. 2019An Improved Analysis of Training Over-parameterized Deep Neural Networkscited by 120
  6. 2020Neural Contextual Bandits with Deep Representation and Shallow Explorationcited by 71
  7. 2019Randomized Exploration in Generalized Linear Banditscited by 70
  8. 2014Algorithms for multi-armed bandit problemscited by 63
  9. 2017Scalable Generalized Linear Bandits: Online Computation and Hashingcited by 63
  10. 2022Computationally Efficient Horizon-Free Reinforcement Learning for Linear Mixture MDPscited by 52
  11. 2021Improved Variance-Aware Confidence Sets for Linear Bandits and Linear Mixture MDPcited by 45
  12. 2021Improved Regret Analysis for Variance-Adaptive Linear Bandits and Horizon-Free Linear Mixture MDPscited by 44
  13. 2021EE-Net: Exploitation-Exploration Neural Networks in Contextual Banditscited by 43
  14. 2023Variance-Dependent Regret Bounds for Linear Bandits and Reinforcement Learning: Adaptivity and Computational Efficiencycited by 39
  15. 2022Corruption-Robust Algorithms with Uncertainty Weighting for Nonlinear Contextual Bandits and Markov Decision Processescited by 33
  16. 2022Langevin Monte Carlo for Contextual Banditscited by 22
  17. 2013Optimal computational and statistical rates of convergence for sparse nonconvex learning problemscited by 14
  18. 2023Density-Softmax: Efficient Test-time Model for Uncertainty Estimation and Robustness under Distribution Shiftscited by 12
  19. 2015Random Forest for the Contextual Bandit Problem - extended versioncited by 11
  20. 2021Online Calibrated and Conformal Prediction Improves Bayesian Optimizationcited by 10
  21. 2024Density-Regression: Efficient and Distance-Aware Deep Regressor for Uncertainty Estimation under Distribution Shiftscited by 7
  22. 2017Speeding Up Latent Variable Gaussian Graphical Model Estimation via Nonconvex Optimizationscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Neural Variance-aware Dueling Bandits with Deep Representation and Shallow Explorationcited by 2
  2. 2025Q-Learning with Shift-Aware Upper Confidence Bound in Non-Stationary Reinforcement Learningcited by 1
  3. 2026Efficient and Uncertainty-Aware Diffusion Framework for Offline-to-Online Reinforcement Learningcited by 0