# Sequential Off-Policy Learning with Logarithmic Smoothing

arXiv: https://arxiv.org/abs/2506.10664 · HTML: https://mlsanity.com/paper/2506.10664/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 62 references that are in the map.
- First posted on arXiv 12 Jun 2025.

## Builds on (most cited first, 25 of 25)

- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data](https://mlsanity.com/paper/1703.11008/) (2017)
- [Is Pessimism Provably Efficient for Offline RL?](https://mlsanity.com/paper/2012.15085/) (2020)
- [User-friendly introduction to PAC-Bayes bounds](https://mlsanity.com/paper/2110.11216/) (2021)
- [Doubly Robust Policy Evaluation and Optimization](https://mlsanity.com/paper/1503.02834/) (2015)
- [A Primer on PAC-Bayesian Learning](https://mlsanity.com/paper/1901.05353/) (2019)
- [Tighter risk certificates for neural networks](https://mlsanity.com/paper/2007.12911/) (2020)
- [Doubly robust off-policy evaluation with shrinkage](https://mlsanity.com/paper/1907.09623/) (2019)
- [Generalization Bounds of SGLD for Non-convex Learning: Two Theoretical Viewpoints](https://mlsanity.com/paper/1707.05947/) (2017)
- [Policy Evaluation and Optimization with Continuous Treatments](https://mlsanity.com/paper/1802.06037/) (2018)
- [Fast Rates for Contextual Linear Optimization](https://mlsanity.com/paper/2011.03030/) (2020)
- [Bayesian Counterfactual Risk Minimization](https://mlsanity.com/paper/1806.11500/) (2018)
- [A unified recipe for deriving (time-uniform) PAC-Bayes bounds](https://mlsanity.com/paper/2302.03421/) (2023)
- [Online PAC-Bayes Learning](https://mlsanity.com/paper/2206.00024/) (2022)
- [PAC-Bayesian Contrastive Unsupervised Representation Learning](https://mlsanity.com/paper/1910.04464/) (2019)
- [PAC-Bayesian Offline Contextual Bandits With Guarantees](https://mlsanity.com/paper/2210.13132/) (2022)
- [BLOB : A Probabilistic Model for Recommendation that Combines Organic and Bandit Signals](https://mlsanity.com/paper/2008.12504/) (2020)
- [Logarithmic Smoothing for Pessimistic Off-Policy Evaluation, Selection and Learning](https://mlsanity.com/paper/2405.14335/) (2024)
- [Exponential Smoothing for Off-Policy Learning](https://mlsanity.com/paper/2305.15877/) (2023)
- [Importance-Weighted Offline Learning Done Right](https://mlsanity.com/paper/2309.15771/) (2023)
- [Topological Generalization Bounds for Discrete-Time Stochastic Optimization Algorithms](https://mlsanity.com/paper/2407.08723/) (2024)
- [Risk Minimization from Adaptively Collected Data: Guarantees for Supervised and Policy Learning](https://mlsanity.com/paper/2106.01723/) (2021)
- [Sequential Counterfactual Risk Minimization](https://mlsanity.com/paper/2302.12120/) (2023)
- [Fast Offline Policy Optimization for Large Scale Recommendation](https://mlsanity.com/paper/2208.05327/) (2022)
- [Unified PAC-Bayesian Study of Pessimism for Offline Policy Learning with Regularized Importance Sampling](https://mlsanity.com/paper/2406.03434/) (2024)

## Abstract (arXiv)

Off-policy learning enables training policies from logged interaction data. Most prior work considers the batch setting, where a policy is learned from data generated by a single behavior policy. In real systems, however, policies are updated and redeployed repeatedly, each time training on all previously collected data while generating new interactions for future updates. This sequential off-policy learning setting is common in practice but remains largely unexplored theoretically. In this work, we present and study a simple algorithm for sequential off-policy learning, combining Logarithmic Smoothing (LS) estimation with online PAC-Bayesian tools. We further show that a principled adjustment to LS improves performance and accelerates convergence under mild conditions. The algorithms introduced generalise previous work: they match state-of-the-art offline approaches in the batch case and substantially outperform them when policies are updated sequentially. Empirical evaluations highlight both the benefits of the sequential framework and the strength of the proposed algorithms.