mlsanityData to 7 Oct 2026

Pessimism for Offline Linear Contextual Bandits using ℓp Confidence Sets

arXiv 2205.10671 ↗First posted 21 May 2022cs.LG stat.MLcited by 18

Not on the map. Why?

Abstract (arXiv)

We present a family hatπp≥ 1 of pessimistic learning rules for offline learning of linear contextual bandits, relying on confidence sets with respect to different ℓp norms, where hatπ2 corresponds to Bellman-consistent pessimism (BCP), while hatπ_∞ is a novel generalization of lower confidence bound (LCB) to the linear setting. We show that the novel hatπ_∞ learning rule is, in a sense, adaptively optimal, as it achieves the minimax performance (up to log factors) against all ℓq-constrained problems, and as such it strictly dominates all other predictors in the family, including hatπ2.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 36 of its 47 references that are in the map. First posted on arXiv 21 May 2022.

Builds on

36 of 47 references.

References · 36

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  4. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  5. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  6. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  7. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  8. 2019A Survey of Autonomous Driving: Common Practices and Emerging Technologiescited by 373
  9. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  10. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  11. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  12. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 275
24 more
  1. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
  2. 2021The Statistical Complexity of Interactive Decision Makingcited by 208
  3. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 160
  4. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learningcited by 157
  5. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?cited by 135
  6. 2021Representation Learning for Online and Offline RL in Low-rank MDPscited by 128
  7. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrabilitycited by 122
  8. 2020Naive Exploration is Optimal for Online LQRcited by 111
  9. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learningcited by 111
  10. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learningcited by 109
  11. 2022Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexitycited by 84
  12. 2022Settling the Sample Complexity of Model-Based Offline Reinforcement Learningcited by 82
  13. 2020Instance-Dependent Complexity of Contextual Bandits and Reinforcement Learning: A Disagreement-Based Perspectivecited by 78
  14. 2021Towards Instance-Optimal Offline Reinforcement Learning with Pessimismcited by 78
  15. 2020Provably Good Batch Reinforcement Learning Without Great Explorationcited by 77
  16. 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximationcited by 69
  17. 2022Near-optimal Offline Reinforcement Learning with Linear Representation: Leveraging Variance Information with Pessimismcited by 63
  18. 2020Exponential Lower Bounds for Batch Reinforcement Learning: Batch RL can be Exponentially Harder than Online RLcited by 55
  19. 2021Risk Bounds and Rademacher Complexity in Batch Reinforcement Learningcited by 42
  20. 2020Is Temporal Difference Learning Optimal? An Instance-Dependent Analysiscited by 41
  21. 2022The Efficacy of Pessimism in Asynchronous Q-Learningcited by 41
  22. 2022Offline Reinforcement Learning Under Value and Density-Ratio Realizability: The Power of Gapscited by 34
  23. 2021On the Optimality of Batch Policy Optimization Algorithmscited by 28
  24. 2021Instance-optimality in optimal value estimation: Adaptivity via variance-reduced Q-learningcited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 7; 2024: 1; 2025: 7; 2026: 2.

18 papers.

Sort:

By area:

  1. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 244
  2. 2023Importance-Weighted Offline Learning Done Rightcited by 15
  3. 2023Oracle-Efficient Pessimism: Offline Policy Optimization in Contextual Banditscited by 14
  4. 2023Differentially Private Reward Estimation with Preference Feedbackcited by 12
  5. 2023Fast and Regret Optimal Best Arm Identification: Fundamental Limits and Low-Complexity Algorithmscited by 10
  6. 2022Multi-Task Off-Policy Learning from Bandit Feedbackcited by 9
  7. 2023Offline Minimax Soft-Q-learning Under Realizability and Partial Coveragecited by 9
  8. 2025Towards a Sharp Analysis of Offline Policy Learning for f-Divergence-Regularized Contextual Banditscited by 9
  9. 2024Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Explorationcited by 6
  10. 2025Offline Learning for Combinatorial Multi-armed Banditscited by 5
  11. 2023Unified Off-Policy Learning to Rank: a Reinforcement Learning Perspectivecited by 4
  12. 2025Balancing optimism and pessimism in offline-to-online learningcited by 4
6 more
  1. 2025Augmenting Online RL with Offline Data is All You Need: A Unified Hybrid RL Algorithm Design and Analysiscited by 3
  2. 2025KL-regularization Itself is Differentially Private in Bandits and RLHFcited by 2
  3. 2025Hybrid Combinatorial Multi-armed Bandits with Probabilistically Triggered Armscited by 2
  4. 2025Group-Sensitive Offline Contextual Banditscited by 0
  5. 2026Unlearning Offline Stochastic Multi-Armed Banditscited by 0
  6. 2026Offline-to-Online Learning in Linear Banditscited by 0