mlsanityData to 7 Oct 2026

Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning

arXiv 2106.04895 ↗First posted 9 Jun 2021cs.LG stat.MLcited by 157

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Recent theoretical work studies sample-efficient reinforcement learning (RL) extensively in two settings: learning interactively in the environment (online RL), or learning from an offline dataset (offline RL). However, existing algorithms and theories for learning near-optimal policies in these two settings are rather different and disconnected. Towards bridging this gap, this paper initiates the theoretical study of policy finetuning, that is, online RL where the learner has additional access to a "reference policy" μ close to the optimal policy π_star in a certain sense. We consider the policy finetuning problem in episodic Markov Decision Processes (MDPs) with S states, A actions, and horizon length H. We first design a sharp offline reduction algorithm -- which simply executes μ and runs offline policy optimization on the collected dataset -- that finds an varepsilon near-optimal policy within widetildeO(H3SC^star/varepsilon2) episodes, where C^star is the single-policy concentrability coefficient between μ and π_star. This offline result is the first that matches the sample complexity lower bound in this setting, and resolves a recent open question in offline RL. We then establish an Omega(H3SminC^star, A/varepsilon2) sample complexity lower bound for any policy finetuning algorithm, including those that can adaptively explore the environment. This implies that -- perhaps surprisingly -- the optimal policy finetuning algorithm is either offline reduction or a purely online RL algorithm that does not use μ. Finally, we design a new hybrid offline/online algorithm for policy finetuning that achieves better sample complexity than both vanilla offline reduction and purely online RL algorithms, in a relaxed setting where μ only satisfies concentrability partially up to a certain time step.

Key facts. Cited by 157 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 43 of its 69 references that are in the map. First posted on arXiv 9 Jun 2021.

Builds on

43 of 69 references.

Closest ancestors · 11

Same direction, newest first

  1. 2021Nearly Horizon-Free Offline Reinforcement Learning25 Mar
  2. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
  3. 2021Near-Optimal Offline Reinforcement Learning via Double Variance Reduction2 Feb
  4. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  5. 2020Provably Good Batch Reinforcement Learning Without Great Exploration16 Jul
  6. 2020Near-Optimal Provable Uniform Convergence in Offline Policy Evaluation for Reinforcement Learning7 Jul
  7. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  8. 2020MOPO: Model-based Offline Policy Optimization27 May
  9. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  10. 2020Q* Approximation Schemes for Batch Reinforcement Learning: A Theoretical Comparison9 Mar
  11. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 32

  1. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  2. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  3. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  4. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  5. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  6. 2019Solving Rubik's Cube with a Robot Handcited by 699
  7. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
  8. 2017Minimax Regret Bounds for Reinforcement Learningcited by 500
  9. 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 494
  10. 2018Is Q-learning Provably Efficient?cited by 485
  11. 2019Emergent Tool Use From Multi-Agent Autocurriculacited by 308
  12. 2016Contextual Decision Processes with Low Bellman Rank are PAC-Learnablecited by 290
20 more
  1. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  2. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 232
  3. 2021Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithmscited by 222
  4. 2020Learning Near Optimal Policies with Low Inherent Bellman Errorcited by 180
  5. 2021Bilinear Classes: A Structural Framework for Provable Generalization in RLcited by 176
  6. 2017Unifying PAC and Regret: Uniform PAC Bounds for Episodic Reinforcement Learningcited by 175
  7. 2020Almost Optimal Model-Free Reinforcement Learning via Reference-Advantage Decompositioncited by 127
  8. 2021Provably Efficient Reinforcement Learning with Linear Function Approximation Under Adaptivity Constraintscited by 124
  9. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  10. 2014Model-based Reinforcement Learning and the Eluder Dimensioncited by 116
  11. 2020Near-Optimal Reinforcement Learning with Self-Playcited by 108
  12. 2020A Sharp Analysis of Model-based Reinforcement Learning with Self-Playcited by 104
  13. 2020Episodic Reinforcement Learning in Finite MDPs: Minimax Lower Bounds Revisitedcited by 104
  14. 2018Policy Certificates: Towards Accountable Reinforcement Learningcited by 81
  15. 2019Provably Efficient Q-Learning with Low Switching Costcited by 77
  16. 2020Provably Efficient Reward-Agnostic Navigation with Linear Value Iterationcited by 56
  17. 2021A Provably Efficient Algorithm for Linear Markov Decision Process with Low Switching Costcited by 38
  18. 2020On Function Approximation in Reinforcement Learning: Optimism in the Face of Large State Spacescited by 19
  19. 2019Benchmarking Bonus-Based Exploration Methods on the Arcade Learning Environmentcited by 17
  20. 2021MUSBO: Model-based Uncertainty Regularized and Sample Efficient Batch Optimization for Deployment Constrained Reinforcement Learningcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 31; 2023: 43; 2024: 32; 2025: 26; 2026: 19.

157 papers.

Sort:

By area:

  1. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  2. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  3. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  4. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 244
  5. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  6. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  7. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
  8. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
  9. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  10. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 103
  11. 2022Jump-Start Reinforcement Learningcited by 102
  12. 2025Reinforcement Learning with Action Chunkingcited by 97
38 more
  1. 2022Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexitycited by 85
  2. 2022Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexitycited by 84
  3. 2022The Role of Coverage in Online Reinforcement Learningcited by 82
  4. 2021Towards Instance-Optimal Offline Reinforcement Learning with Pessimismcited by 78
  5. 2023Constrained Decision Transformer for Offline Safe Reinforcement Learningcited by 65
  6. 2022Near-optimal Offline Reinforcement Learning with Linear Representation: Leveraging Variance Information with Pessimismcited by 63
  7. 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Datacited by 58
  8. 2024The Importance of Online Data: Understanding Preference Fine-tuning via Coveragecited by 55
  9. 2022Nearly Minimax Optimal Offline Reinforcement Learning with Linear Function Approximation: Single-Agent MDP and Markov Gamecited by 53
  10. 2023Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coveragecited by 51
  11. 2023The Curious Price of Distributional Robustness in Reinforcement Learning with a Generative Modelcited by 50
  12. 2023Settling the Sample Complexity of Online Reinforcement Learningcited by 44
  13. 2022Leveraging Offline Data in Online Reinforcement Learningcited by 42
  14. 2023ChiPFormer: Transferable Chip Placement via Offline Decision Transformercited by 42
  15. 2022The Efficacy of Pessimism in Asynchronous Q-Learningcited by 41
  16. 2022Offline Reinforcement Learning Under Value and Density-Ratio Realizability: The Power of Gapscited by 34
  17. 2026Q-learning with Adjoint Matchingcited by 33
  18. 2022Policy learning "without" overlap: Pessimism and generalized empirical Bernstein's inequalitycited by 32
  19. 2022When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learningcited by 31
  20. 2023Adversarial Model for Offline Reinforcement Learningcited by 31
  21. 2022Provably Efficient Offline Multi-agent Reinforcement Learning via Strategy-wise Bonuscited by 29
  22. 2022Provable Benefit of Multitask Representation Learning in Reinforcement Learningcited by 29
  23. 2022When is Offline Two-Player Zero-Sum Markov Game Solvable?cited by 27
  24. 2022Towards Deployment-Efficient Reinforcement Learning: Lower Bound and Optimalitycited by 24
  25. 2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruptioncited by 24
  26. 2023RLIF: Interactive Imitation Learning as Reinforcement Learningcited by 24
  27. 2023Accelerating Exploration with Unlabeled Prior Datacited by 21
  28. 2022Offline Reinforcement Learning with Differential Privacycited by 19
  29. 2025VEM: Environment-Free Exploration for Training GUI Agent with Value Environment Modelcited by 19
  30. 2022Pessimism for Offline Linear Contextual Bandits using ℓp Confidence Setscited by 18
  31. 2023Offline Learning in Markov Games with General Function Approximationcited by 16
  32. 2023High-probability sample complexities for policy evaluation with linear function approximationcited by 16
  33. 2023B-Coder: Value-Based Deep Reinforcement Learning for Program Synthesiscited by 16
  34. 2023Reward-agnostic Fine-tuning: Provable Statistical Benefits of Hybrid Reinforcement Learningcited by 15
  35. 2022A Self-Play Posterior Sampling Algorithm for Zero-Sum Markov Gamescited by 14
  36. 2023Minimax-Optimal Reward-Agnostic Exploration in Reinforcement Learningcited by 14
  37. 2024Offline Multitask Representation Learning for Reinforcement Learningcited by 14
  38. 2024Bayesian Design Principles for Offline-to-Online Reinforcement Learningcited by 14

50 most cited of 157 All 157 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions
  2. 2022Settling the Sample Complexity of Model-Based Offline Reinforcement Learning
  3. 2025Optimal Single-Policy Sample Complexity and Transient Coverage for Average-Reward Offline RL
  4. 2023On the Sample Complexity of Vanilla Model-Based Offline Reinforcement Learning with Dependent Samples
  5. 2022Offline Reinforcement Learning via Linear-Programming with Error-Bound Induced Constraints
  6. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond
  7. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrability
  8. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets