mlsanityData to 7 Oct 2026

Oracle-Efficient Pessimism: Offline Policy Optimization in Contextual Bandits

arXiv 2306.07923 ↗First posted 13 Jun 2023cs.LGcited by 14

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

We consider offline policy optimization (OPO) in contextual bandits, where one is given a fixed dataset of logged interactions. While pessimistic regularizers are typically used to mitigate distribution shift, prior implementations thereof are either specialized or computationally inefficient. We present the first general oracle-efficient algorithm for pessimistic OPO: it reduces to supervised learning, leading to broad applicability. We obtain statistical guarantees analogous to those for prior pessimistic approaches. We instantiate our approach for both discrete and continuous actions and perform experiments in both settings, showing advantage over unregularized OPO across a wide range of configurations.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 39 of its 64 references that are in the map. First posted on arXiv 13 Jun 2023.

Builds on

39 of 64 references.

Closest ancestors · 11

Same direction, newest first

  1. 2022Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangian1 Nov
  2. 2021Offline Neural Contextual Bandits: Pessimism, Optimization and Generalization27 Nov
  3. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage13 Jul
  4. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  5. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
  6. 2021Instabilities of Offline RL with Pre-Trained Neural Representation8 Mar
  7. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  8. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  9. 2020MOPO: Model-based Offline Policy Optimization27 May
  10. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  11. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May

Other references · 28

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  3. 2014OpenML: networked science in machine learningcited by 523
  4. 2019Introduction to Multi-Armed Banditscited by 475
  5. 2015Recursive Partitioning for Heterogeneous Causal Effectscited by 254
  6. 2014Taming the Monster: A Fast and Simple Algorithm for Contextual Banditscited by 200
  7. 2015Doubly Robust Policy Evaluation and Optimizationcited by 158
  8. 2018More Robust Doubly Robust Off-policy Evaluationcited by 141
  9. 2016Optimal and Adaptive Off-policy Evaluation in Contextual Banditscited by 111
  10. 2018Practical Contextual Bandits with Regression Oraclescited by 92
  11. 2019Doubly robust off-policy evaluation with shrinkagecited by 87
  12. 2021Conservative Objective Models for Effective Offline Model-Based Optimizationcited by 87
16 more
  1. 2018A Contextual Bandit Bake-offcited by 64
  2. 2018Policy Evaluation and Optimization with Continuous Treatmentscited by 57
  3. 2017Balanced Policy Evaluation and Learningcited by 51
  4. 2022Smoothed Online Learning is as Easy as Statistical Learningcited by 50
  5. 2019Contextual Bandits with Continuous Actions: Smoothing, Zooming, and Adaptingcited by 42
  6. 2022Contextual Bandits with Large Action Spaces: Made Practicalcited by 32
  7. 2022Policy learning "without" overlap: Pessimism and generalized empirical Bernstein's inequalitycited by 32
  8. 2018Bayesian Counterfactual Risk Minimizationcited by 28
  9. 2019Semi-Parametric Efficient Policy Learning with Continuous Actionscited by 26
  10. 2020Efficient Contextual Bandits with Continuous Actionscited by 24
  11. 2022Oracle-Efficient Online Learning for Beyond Worst-Case Adversariescited by 19
  12. 2022PAC-Bayesian Offline Contextual Bandits With Guaranteescited by 19
  13. 2022Pessimism for Offline Linear Contextual Bandits using ℓp Confidence Setscited by 18
  14. 2022Contextual Bandits with Smooth Regret: Efficient Learning in Continuous Action Spacescited by 17
  15. 2023Exponential Smoothing for Off-Policy Learningcited by 15
  16. 2022Boosted Off-Policy Learningcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 4; 2025: 5; 2026: 3.

14 papers.

Sort:

By area:

  1. 2024Logarithmic Smoothing for Pessimistic Off-Policy Evaluation, Selection and Learningcited by 16
  2. 2023Importance-Weighted Offline Learning Done Rightcited by 15
  3. 2023Off-Policy Evaluation for Large Action Spaces via Policy Convolutioncited by 14
  4. 2024Bayesian Off-Policy Evaluation and Learning for Large Action Spacescited by 5
  5. 2024Unified PAC-Bayesian Study of Pessimism for Offline Policy Learning with Regularized Importance Samplingcited by 3
  6. 2024Off-policy estimation with adaptively collected data: the power of online learningcited by 3
  7. 2025Improved Offline Contextual Bandits with Second-Order Bounds: Betting and Freezingcited by 3
  8. 2025Augmenting Online RL with Offline Data is All You Need: A Unified Hybrid RL Algorithm Design and Analysiscited by 3
  9. 2025SquarechiPO: Differentially Private and Robust chi2-Preference Optimization in Offline Direct Alignmentcited by 3
  10. 2026When Does Online Imitation Learning Help in LLM Post-Training? The Role of (Non-)Realizability Beyond Horizoncited by 2
  11. 2026Direction-Aware Offline-to-Online Learning in Linear Contextual Banditscited by 1
  12. 2025Clustered KL-barycenter design for policy evaluationcited by 0
2 more
  1. 2025Offline Clustering of Preference Learning with Active-data Augmentationcited by 0
  2. 2026Efficient Exploration for Iterative Nash Preference Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Adapting to Misspecification in Contextual Bandits with Offline Regression Oracles
  2. 2020POPO: Pessimistic Offline Policy Optimization
  3. 2022State-Aware Proximal Pessimistic Algorithms for Offline Reinforcement Learning
  4. 2022Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processes
  5. 2023Contextual Bandits and Imitation Learning via Preference-Based Active Queries
  6. 2021Dealing with the Unknown: Pessimistic Offline Reinforcement Learning
  7. 2025Provable Zero-Shot Generalization in Offline Reinforcement Learning
  8. 2023Kernelized Offline Contextual Dueling Bandits