mlsanityData to 14 Oct 2026

Statistically Efficient Advantage Learning for Offline Reinforcement Learning in Infinite Horizons

arXiv 2202.13163 ↗First posted 26 Feb 2022stat.ML cs.LGcited by 10

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

We consider reinforcement learning (RL) methods in offline domains without additional online data collection, such as mobile health applications. Most of existing policy optimization algorithms in the computer science literature are developed in online settings where data are easy to collect or simulate. Their generalizations to mobile health applications with a pre-collected offline dataset remain unknown. The aim of this paper is to develop a novel advantage learning framework in order to efficiently use pre-collected data for policy optimization. The proposed method takes an optimal Q-estimator computed by any existing state-of-the-art RL algorithms as input, and outputs a new policy whose value is guaranteed to converge at a faster rate than the policy derived based on the initial Q-estimator. Extensive numerical experiments are conducted to back up our theoretical findings. A Python implementation of our proposed method is available at https://github.com/leyuanheart/SEAL.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 37 of its 106 references that are in the map. First posted on arXiv 26 Feb 2022.

Builds on

37 of 106 references.

Closest ancestors · 3

Same direction, newest first

  1. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  2. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May

Other references · 34

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  4. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  5. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  6. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  7. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
  8. 2017Quasi-Oracle Estimation of Heterogeneous Treatment Effectscited by 332
  9. 2019Causal Confusion in Imitation Learningcited by 288
  10. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 286
  11. 2019Batch Policy Learning under Constraintscited by 244
  12. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 227
22 more
  1. 2019A Theoretical Analysis of Deep Q-Learningcited by 215
  2. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  3. 2019Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processescited by 119
  4. 2020Sample Complexity of Asynchronous Q-Learning: Sharper Analysis and Variance Reductioncited by 95
  5. 2021Is Q-Learning Minimax Optimal? A Tight Sample Complexity Analysiscited by 92
  6. 2019Finite-Sample Analysis for SARSA with Linear Function Approximationcited by 85
  7. 2019Variance-reduced Q-learning is minimax optimalcited by 73
  8. 2020Batch Policy Learning in Average Reward Markov Decision Processescited by 69
  9. 2016Estimating Dynamic Treatment Regimes in Mobile Health Using V-learningcited by 65
  10. 2018Real-Time Bidding with Multi-Agent Reinforcement Learning in Display Advertisingcited by 65
  11. 2020Statistical Inference of the Value Function for Reinforcement Learning in Infinite Horizon Settingscited by 65
  12. 2018Policy Evaluation and Optimization with Continuous Treatmentscited by 58
  13. 2019Learning When-to-Treat Policiescited by 49
  14. 2021Deeply-Debiased Off-Policy Interval Estimationcited by 36
  15. 2022On Well-posedness and Minimax Optimal Rates of Nonparametric Q-function Estimation in Off-policy Evaluationcited by 33
  16. 2020Does the Markov Decision Process Fit the Data: Testing for the Markov Property in Sequential Decision Makingcited by 30
  17. 2021Estimating Optimal Infinite Horizon Dynamic Treatment Regimes via pT-Learningcited by 26
  18. 2016Interpretable Dynamic Treatment Regimescited by 17
  19. 2020Personalized Policy Learning using Longitudinal Mobile Health Datacited by 10
  20. 2021Jump Interval-Learning for Individualized Decision Makingcited by 5
  21. 2014Constructing Dynamic Treatment Regimes in Infinite-Horizon Settingscited by 4
  22. 2021Robust Estimation of Heterogeneous Treatment Effects using Electronic Health Record Datacited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 2; 2024: 0; 2025: 3; 2026: 3.

10 papers.

  1. 2022Offline Reinforcement Learning for Safer Blood Glucose Control in People with Type 1 Diabetescited by 23
  2. 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statisticcited by 21
  3. 2022Testing Stationarity and Change Point Detection in Reinforcement Learningcited by 12
  4. 2025Doubly Robust Alignment for Large Language Modelscited by 12
  5. 2023Quasi-optimal Reinforcement Learning with Continuous Actionscited by 7
  6. 2023Inference on Optimal Dynamic Policies via Softmax Approximationcited by 3
  7. 2025Statistical Inference in Reinforcement Learning: A Selective Surveycited by 2
  8. 2025Generalized Fitted Q-Iteration with Clustered Datacited by 0
  9. 2026Counterfactually Safe Reinforcement Learningcited by 0
  10. 2026Primal--Dual Alternating Neural Learning for Timely Classification with Performance Guaranteescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond
  2. 2023Offline Policy Evaluation for Reinforcement Learning with Adaptively Collected Data
  3. 2020MOPO: Model-based Offline Policy Optimization
  4. 2020MOReL : Model-Based Offline Reinforcement Learning
  5. 2025On the Statistical Complexity for Offline and Low-Adaptive Reinforcement Learning with Structures
  6. 2025Online Optimization for Offline Safe Reinforcement Learning
  7. 2022Oracle Inequalities for Model Selection in Offline Reinforcement Learning
  8. 2025Augmenting Online RL with Offline Data is All You Need: A Unified Hybrid RL Algorithm Design and Analysis