mlsanityData to 14 Oct 2026

Offline Neural Contextual Bandits: Pessimism, Optimization and Generalization

arXiv 2111.13807 ↗First posted 27 Nov 2021cs.LG cs.AI stat.MLcited by 25

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline policy learning (OPL) leverages existing data collected a priori for policy optimization without any active exploration. Despite the prevalence and recent interest in this problem, its theoretical and algorithmic foundations in function approximation settings remain under-developed. In this paper, we consider this problem on the axes of distributional shift, optimization, and generalization in offline contextual bandits with neural networks. In particular, we propose a provably efficient offline contextual bandit with neural network function approximation that does not require any functional assumption on the reward. We show that our method provably generalizes over unseen contexts under a milder condition for distributional shift than the existing OPL works. Notably, unlike any other OPL method, our method learns from the offline data in an online manner using stochastic gradient descent, allowing us to leverage the benefits of online learning into an offline setting. Moreover, we show that our method is more computationally efficient and has a better dependence on the effective dimension of the neural network than an online counterpart. Finally, we demonstrate the empirical effectiveness of our method in a range of synthetic and real-world OPL problems.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 33 of its 59 references that are in the map. First posted on arXiv 27 Nov 2021.

Builds on

33 of 59 references.

Closest ancestors · 6

Same direction, newest first

  1. 2021Risk Bounds and Rademacher Complexity in Batch Reinforcement Learning25 Mar
  2. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
  3. 2021Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiency5 Feb
  4. 2021Fast Rates for the Regret of Offline Reinforcement Learning31 Jan
  5. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May

Other references · 27

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2016Layer Normalizationcited by 5,669
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  5. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  6. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  7. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  8. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  9. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
  10. 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 222
  11. 2014Taming the Monster: A Fast and Simple Algorithm for Contextual Banditscited by 201
  12. 2019Reinforcement Learning in Feature Space: Matrix Bandit, Kernels, and Regret Boundcited by 201
15 more
  1. 2013Finite-Time Analysis of Kernelised Contextual Banditscited by 177
  2. 2018Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Samplingcited by 169
  3. 2021Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolationcited by 132
  4. 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
  5. 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 115
  6. 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
  7. 2020Neural Contextual Bandits with Deep Representation and Shallow Explorationcited by 71
  8. 2020Asymptotically Efficient Off-Policy Evaluation for Tabular Reinforcement Learningcited by 60
  9. 2019Learning When-to-Treat Policiescited by 49
  10. 2021On the Proof of Global Convergence of Gradient Descent for Deep ReLU Networks with Linear Widthscited by 44
  11. 2021Off-Policy Evaluation via Adaptive Weighting with Data from Contextual Banditscited by 40
  12. 2021Leveraging Good Representations in Linear Contextual Banditscited by 29
  13. 2021On the Optimality of Batch Policy Optimization Algorithmscited by 28
  14. 2020Offline Contextual Bandits with Overparameterized Modelscited by 10
  15. 2021Combining Online Learning and Offline Learning for Contextual Bandits with Deficient Supportcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 6; 2023: 6; 2024: 5; 2025: 7; 2026: 1.

25 papers.

Sort:

By area:

  1. 2024An Overview of Diffusion Models: Applications, Guided Generation, Statistical Rates and Optimizationcited by 73
  2. 2023Reward-Directed Conditional Diffusion: Provable Distribution Estimation and Reward Improvementcited by 49
  3. 2022Data-Driven Offline Decision-Making via Invariant Representation Learningcited by 31
  4. 2024Diffusion Model for Data-Driven Black-Box Optimizationcited by 24
  5. 2022On Instance-Dependent Bounds for Offline Reinforcement Learning with Linear Function Approximationcited by 22
  6. 2022PAC-Bayesian Offline Contextual Bandits With Guaranteescited by 20
  7. 2022Federated Neural Banditscited by 17
  8. 2023On the Convergence and Sample Complexity Analysis of Deep Q-Networks with ε-Greedy Explorationcited by 17
  9. 2023Oracle-Efficient Pessimism: Offline Policy Optimization in Contextual Banditscited by 15
  10. 2023Optimal Best-Arm Identification in Bandits with Access to Offline Datacited by 8
  11. 2023On the Disconnect Between Theory and Practice of Neural Networks: Limits of the NTK Perspectivecited by 6
  12. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyondcited by 6
13 more
  1. 2024Learning Decision Policies with Instrumental Variables through Double Machine Learningcited by 5
  2. 2025Offline Learning for Combinatorial Multi-armed Banditscited by 5
  3. 2023VIPeR: Provably Efficient Algorithm for Offline RL with Neural Function Approximationcited by 3
  4. 2025Online Clustering of Dueling Banditscited by 3
  5. 2025Augmenting Online RL with Offline Data is All You Need: A Unified Hybrid RL Algorithm Design and Analysiscited by 3
  6. 2025Provable Anytime Ensemble Sampling Algorithms in Nonlinear Contextual Banditscited by 2
  7. 2025Hybrid Combinatorial Multi-armed Bandits with Probabilistically Triggered Armscited by 2
  8. 2022Text Summarization with Oracle Expectationcited by 1
  9. 2024Graph Neural Thompson Samplingcited by 1
  10. 2025NeuroSep-CP-LCB: A Deep Learning-based Contextual Multi-armed Bandit Algorithm with Uncertainty Quantification for Early Sepsis Predictioncited by 1
  11. 2022On Practical Reinforcement Learning: Provable Robustness, Scalability, and Statistical Efficiencycited by 0
  12. 2025Variational Quantum Circuits in Offline Contextual Bandit Problemscited by 0
  13. 2026Diffusion Policy with Bayesian Expert Selection for Active Multi-Target Trackingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Group-Sensitive Offline Contextual Bandits
  2. 2026Offline-to-Online Learning in Linear Bandits
  3. 2026Offline Contextual Bandits in the Presence of New Actions
  4. 2020MOPO: Model-based Offline Policy Optimization
  5. 2026Direction-Aware Offline-to-Online Learning in Linear Contextual Bandits
  6. 2023Selective Uncertainty Propagation in Offline RL
  7. 2023Kernelized Offline Contextual Dueling Bandits
  8. 2025On the Statistical Complexity for Offline and Low-Adaptive Reinforcement Learning with Structures