mlsanityData to 7 Oct 2026

Unified PAC-Bayesian Study of Pessimism for Offline Policy Learning with Regularized Importance Sampling

arXiv 2406.03434 ↗First posted 5 Jun 2024cs.LG cs.AI stat.MLcited by 3

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Off-policy learning (OPL) often involves minimizing a risk estimator based on importance weighting to correct bias from the logging policy used to collect data. However, this method can produce an estimator with a high variance. A common solution is to regularize the importance weights and learn the policy by minimizing an estimator with penalties derived from generalization bounds specific to the estimator. This approach, known as pessimism, has gained recent attention but lacks a unified framework for analysis. To address this gap, we introduce a comprehensive PAC-Bayesian framework to examine pessimism with regularized importance weighting. We derive a tractable PAC-Bayesian generalization bound that universally applies to common importance weight regularizations, enabling their comparison within a single framework. Our empirical results challenge common understanding, demonstrating the effectiveness of standard IW regularization techniques.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 31 of its 55 references that are in the map. First posted on arXiv 5 Jun 2024.

Builds on

31 of 55 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Importance-Weighted Offline Learning Done Right27 Sep
  2. 2023Oracle-Efficient Pessimism: Offline Policy Optimization in Contextual Bandits13 Jun
  3. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec

Other references · 28

  1. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  2. 2015Variational Dropout and the Local Reparameterization Trickcited by 489
  3. 2017A Tutorial on Thompson Samplingcited by 420
  4. 2021User-friendly introduction to PAC-Bayes boundscited by 234
  5. 2014Taming the Monster: A Fast and Simple Algorithm for Contextual Banditscited by 200
  6. 2015Doubly Robust Policy Evaluation and Optimizationcited by 158
  7. 2018More Robust Doubly Robust Off-policy Evaluationcited by 141
  8. 2016Optimal and Adaptive Off-policy Evaluation in Contextual Banditscited by 111
  9. 2016Off-policy evaluation for slate recommendationcited by 102
  10. 2018Offline A/B testing for Recommender Systemscited by 94
  11. 2019Doubly robust off-policy evaluation with shrinkagecited by 87
  12. 2020Off-policy Bandits with Deficient Supportcited by 60
16 more
  1. 2022Off-Policy Evaluation for Large Action Spaces via Embeddingscited by 53
  2. 2016Cascading Bandits for Large-Scale Recommendation Problemscited by 45
  3. 2020Confident Off-Policy Evaluation and Selection through Self-Normalized Importance Weightingcited by 38
  4. 2023Off-Policy Evaluation for Large Action Spaces via Conjunct Effect Modelingcited by 34
  5. 2022Contextual Bandits with Large Action Spaces: Made Practicalcited by 32
  6. 2020Optimal Off-Policy Evaluation from Multiple Logging Policiescited by 30
  7. 2018Bayesian Counterfactual Risk Minimizationcited by 28
  8. 2019Distributionally Robust Counterfactual Risk Minimizationcited by 26
  9. 2020BLOB : A Probabilistic Model for Recommendation that Combines Organic and Bandit Signalscited by 19
  10. 2022PAC-Bayesian Offline Contextual Bandits With Guaranteescited by 19
  11. 2022Deep Hierarchy in Banditscited by 18
  12. 2023Exponential Smoothing for Off-Policy Learningcited by 15
  13. 2022Mixed-Effect Thompson Samplingcited by 14
  14. 2022Probabilistic Rank and Reward: A Scalable Model for Slate Recommendationcited by 7
  15. 2024Diffusion Models Meet Contextual Banditscited by 5
  16. 2024Bayesian Off-Policy Evaluation and Learning for Large Action Spacescited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Sequential Off-Policy Learning with Logarithmic Smoothingcited by 0
  2. 2025Offline Contextual Bandit with Counterfactual Sample Identificationcited by 0
  3. 2026Certified Policy Optimisation for Nested Causal Bandits via PAC-Bayes Riskcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processes
  2. 2022State-Aware Proximal Pessimistic Algorithms for Offline Reinforcement Learning
  3. 2026Generalization in offline RL: The structure is more important than the amount of pessimism
  4. 2022Offline Policy Optimization with Eligible Actions
  5. 2022Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief
  6. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage
  7. 2022Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexity
  8. 2024Pessimistic Causal Reinforcement Learning with Mediators for Confounded Offline Data