Selective Uncertainty Propagation in Offline RL
On the map: Offline RL and imitation learning from data → Offline RL algorithms
Abstract (arXiv)
We consider the finite-horizon offline reinforcement learning (RL) setting, and are motivated by the challenge of learning the policy at any step h in dynamic programming (DP) algorithms. To learn this, it is sufficient to evaluate the treatment effect of deviating from the behavioral policy at step h after having optimized the policy for all future steps. Since the policy at any step can affect next-state distributions, the related distributional shift challenges can make this problem far more statistically hard than estimating such treatment effects in the stochastic contextual bandit setting. However, the hardness of many real-world RL instances lies between the two regimes. We develop a flexible and general method called selective uncertainty propagation for confidence interval construction that adapts to the hardness of the associated distribution shift challenges. We show benefits of our approach on toy environments and demonstrate the benefits of these techniques for offline policy learning.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 21 of its 45 references that are in the map. First posted on arXiv 1 Feb 2023.
Builds on
21 of 45 references.
Closest ancestors · 5
Same direction, newest first
- 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation21 Nov
- 2021Towards Instance-Optimal Offline Reinforcement Learning with Pessimism17 Oct
- 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
- 2020MOReL : Model-Based Offline Reinforcement Learning12 May
- 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
Other references · 16
- 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
- 2017Meta-learners for Estimating Heterogeneous Treatment Effects using Machine Learningcited by 475
- 2017A Tutorial on Thompson Samplingcited by 420
- 2017Quasi-Oracle Estimation of Heterogeneous Treatment Effectscited by 328
- 2014Taming the Monster: A Fast and Simple Algorithm for Contextual Banditscited by 200
- 2019Tighter Problem-Dependent Regret Bounds in Reinforcement Learning without Domain Knowledge using Value Function Boundscited by 194
- 2015Doubly Robust Policy Evaluation and Optimizationcited by 158
- 2017Deep Exploration via Randomized Value Functionscited by 155
- 2019Orthogonal Statistical Learningcited by 150
- 2021Epistemic Neural Networkscited by 120
- 2019Optimism in Reinforcement Learning with Generalized Linear Function Approximationcited by 111
- 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
4 more
- 2017Count-Based Exploration in Feature Space for Reinforcement Learningcited by 48
- 2018Adversarial Domain Adaptation for Stable Brain-Machine Interfacescited by 27
- 2018Semiparametric Contextual Banditscited by 19
- 2022A Bayesian Approach to Learning Bandit Structure in Markov Decision Processescited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024Diffusion Policies for Risk-Averse Behavior Modeling in Offline Reinforcement Learning
- 2023Bridging Distributionally Robust Learning and Offline RL: An Approach to Mitigate Distribution Shift and Partial Data Coverage
- 2024Minimax Optimal and Computationally Efficient Algorithms for Distributionally Robust Offline Reinforcement Learning
- 2025On the Statistical Complexity for Offline and Low-Adaptive Reinforcement Learning with Structures
- 2023Achieving the Asymptotically Optimal Sample Complexity of Offline Reinforcement Learning: A DRO-Based Approach
- 2020Expert-Supervised Reinforcement Learning for Offline Policy Learning and Evaluation
- 2025Towards Optimal Offline Reinforcement Learning
- 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory