mlsanityData to 7 Oct 2026

Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism

arXiv 2103.12021 ↗First posted 22 Mar 2021cs.LG cs.AI math.OCcited by 275

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline (or batch) reinforcement learning (RL) algorithms seek to learn an optimal policy from a fixed dataset without active data collection. Based on the composition of the offline dataset, two main categories of methods are used: imitation learning which is suitable for expert datasets and vanilla offline RL which often requires uniform coverage datasets. From a practical standpoint, datasets often deviate from these two extremes and the exact data composition is usually unknown a priori. To bridge this gap, we present a new offline RL framework that smoothly interpolates between the two extremes of data composition, hence unifying imitation learning and vanilla offline RL. The new framework is centered around a weak version of the concentrability coefficient that measures the deviation from the behavior policy to the expert policy alone. Under this new framework, we further investigate the question on algorithm design: can one develop an algorithm that achieves a minimax optimal rate and also adapts to unknown data composition? To address this question, we consider a lower confidence bound (LCB) algorithm developed based on pessimism in the face of uncertainty in offline RL. We study finite-sample properties of LCB as well as information-theoretic limits in multi-armed bandits, contextual bandits, and Markov decision processes (MDPs). Our analysis reveals surprising facts about optimality rates. In particular, in all three settings, LCB achieves a faster rate of 1/N for nearly-expert datasets compared to the usual rate of 1/√N in offline RL, where N is the number of samples in the batch dataset. In the case of contextual bandits with at least two contexts, we prove that LCB is adaptively optimal for the entire data composition range, achieving a smooth transition from imitation learning to offline RL. We further show that LCB is almost adaptively optimal in MDPs.

Key facts. Cited by 275 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 62 of its 109 references that are in the map. First posted on arXiv 22 Mar 2021.

Builds on

62 of 109 references.

Closest ancestors · 19

Same direction, newest first

  1. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  2. 2021Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiency5 Feb
  3. 2021Near-Optimal Offline Reinforcement Learning via Double Variance Reduction2 Feb
  4. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  5. 2020Sparse Feature Selection Makes Batch Reinforcement Learning More Sample Efficient8 Nov
  6. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?22 Oct
  7. 2020Batch Value-function Approximation with Only Realizability11 Aug
  8. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
  9. 2020Provably Good Batch Reinforcement Learning Without Great Exploration16 Jul
  10. 2020Near-Optimal Provable Uniform Convergence in Offline Policy Evaluation for Reinforcement Learning7 Jul
  11. 2020Critic Regularized Regression26 Jun
  12. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
7 more
  1. 2020MOPO: Model-based Offline Policy Optimization27 May
  2. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  5. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  6. 2019Behavior Regularized Offline Reinforcement Learning26 Nov
  7. 2019Benchmarking Batch Deep Reinforcement Learning Algorithms3 Oct

Other references · 43

  1. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  2. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  4. 2016End to End Learning for Self-Driving Carscited by 1,000
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  6. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  7. 2018Is Q-learning Provably Efficient?cited by 485
  8. 2019A Survey of Autonomous Driving: Common Practices and Emerging Technologiescited by 373
  9. 2016Contextual Decision Processes with Low Bellman Rank are PAC-Learnablecited by 290
  10. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  11. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  12. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
31 more
  1. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  2. 2019On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shiftcited by 174
  3. 2019Neural Policy Gradient Methods: Global Optimality and Rates of Convergencecited by 157
  4. 2016Variance-based regularization with convex objectivescited by 153
  5. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  6. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 141
  7. 2019Is a Good Representation Sufficient for Sample Efficient Reinforcement Learning?cited by 140
  8. 2019Model-Based Reinforcement Learning with a Generative Model is Minimax Optimalcited by 124
  9. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  10. 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
  11. 2020Breaking the Sample Size Barrier in Model-Based Reinforcement Learning with a Generative Modelcited by 108
  12. 2020Variational Policy Gradient Method for Reinforcement Learning with General Utilitiescited by 108
  13. 2015Sample Complexity of Episodic Fixed-Horizon Reinforcement Learningcited by 105
  14. 2020Episodic Reinforcement Learning in Finite MDPs: Minimax Lower Bounds Revisitedcited by 104
  15. 2018Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendationcited by 101
  16. 2020Reinforcement Learning via Fenchel-Rockafellar Dualitycited by 100
  17. 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
  18. 2020Toward the Fundamental Limits of Imitation Learningcited by 96
  19. 2020Error Bounds of Imitating Policies and Environmentscited by 90
  20. 2018Learning Montezuma's Revenge from a Single Demonstrationcited by 76
  21. 2020GradientDICE: Rethinking Generalized Offline Estimation of Stationary Valuescited by 70
  22. 2020Batch Policy Learning in Average Reward Markov Decision Processescited by 69
  23. 2017Variance Reduced Value Iteration and Faster Algorithms for Solving Markov Decision Processescited by 56
  24. 2020Exponential Lower Bounds for Batch Reinforcement Learning: Batch RL can be Exponentially Harder than Online RLcited by 55
  25. 2019Learning When-to-Treat Policiescited by 49
  26. 2014Approximate Policy Iteration Schemes: A Comparisoncited by 39
  27. 2019A Kernel Loss for Solving the Bellman Equationcited by 37
  28. 2018Near-Optimal Time and Sample Complexities for Solving Discounted Markov Decision Process with a Generative Modelcited by 22
  29. 2019Safe Policy Improvement with Soft Baseline Bootstrappingcited by 22
  30. 2019On Value Functions and the Agent-Environment Boundarycited by 19
  31. 2021Minimax Off-Policy Evaluation for Multi-Armed Banditscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 16; 2022: 59; 2023: 72; 2024: 52; 2025: 41; 2026: 35.

275 papers.

Sort:

By area:

  1. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  2. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 244
  3. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 160
  4. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learningcited by 157
  5. 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 137
  6. 2021Representation Learning for Online and Offline RL in Low-rank MDPscited by 128
  7. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 126
  8. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrabilitycited by 122
  9. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
  10. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learningcited by 109
  11. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  12. 2022Jump-Start Reinforcement Learningcited by 102
38 more
  1. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 92
  2. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?cited by 86
  3. 2022Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexitycited by 85
  4. 2022Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexitycited by 84
  5. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothingcited by 83
  6. 2022Settling the Sample Complexity of Model-Based Offline Reinforcement Learningcited by 82
  7. 2022The Role of Coverage in Online Reinforcement Learningcited by 82
  8. 2023Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretrainingcited by 82
  9. 2025Scaling Test-Time Compute Without Verification or RL is Suboptimalcited by 82
  10. 2021Towards Instance-Optimal Offline Reinforcement Learning with Pessimismcited by 78
  11. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHFcited by 73
  12. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregationcited by 71
  13. 2023Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimismcited by 69
  14. 2022Near-optimal Offline Reinforcement Learning with Linear Representation: Leveraging Variance Information with Pessimismcited by 63
  15. 2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHFcited by 57
  16. 2023Provable Offline Preference-Based Reinforcement Learningcited by 55
  17. 2022Nearly Minimax Optimal Offline Reinforcement Learning with Linear Function Approximation: Single-Agent MDP and Markov Gamecited by 53
  18. 2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Modelcited by 52
  19. 2021Corruption-Robust Offline Reinforcement Learningcited by 51
  20. 2023Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coveragecited by 51
  21. 2023The Curious Price of Distributional Robustness in Reinforcement Learning with a Generative Modelcited by 50
  22. 2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hackingcited by 50
  23. 2022Hierarchical Planning Through Goal-Conditioned Offline Reinforcement Learningcited by 48
  24. 2021Offline Reinforcement Learning as Anti-Explorationcited by 44
  25. 2023Settling the Sample Complexity of Online Reinforcement Learningcited by 44
  26. 2022Pessimistic Minimax Value Iteration: Provably Efficient Equilibrium Learning from Offline Datasetscited by 42
  27. 2022Leveraging Offline Data in Online Reinforcement Learningcited by 42
  28. 2022The Efficacy of Pessimism in Asynchronous Q-Learningcited by 41
  29. 2021MADE: Exploration via Maximizing Deviation from Explored Regionscited by 40
  30. 2021Offline Reinforcement Learning with Value-based Episodic Memorycited by 36
  31. 2023Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learningcited by 36
  32. 2022Offline Reinforcement Learning Under Value and Density-Ratio Realizability: The Power of Gapscited by 34
  33. 2024Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimationcited by 33
  34. 2021Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learningcited by 32
  35. 2022Policy learning "without" overlap: Pessimism and generalized empirical Bernstein's inequalitycited by 32
  36. 2022Distributionally Robust Offline Reinforcement Learning with Linear Function Approximationcited by 31
  37. 2023Adversarial Model for Offline Reinforcement Learningcited by 31
  38. 2023Corruption-Robust Offline Reinforcement Learning with General Function Approximationcited by 30

50 most cited of 275 All 275 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Using Non-Expert Data to Robustify Imitation Learning via Offline Reinforcement Learning
  2. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learning
  3. 2023A Strong Baseline for Batch Imitation Learning
  4. 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments
  5. 2021A Minimalist Approach to Offline Reinforcement Learning
  6. 2021Representation Matters: Offline Pretraining for Sequential Decision Making
  7. 2020Offline Reinforcement Learning Hands-On
  8. 2025Evaluation-Time Policy Switching for Offline Reinforcement Learning