mlsanityData to 7 Oct 2026

Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions

arXiv 2303.17396 ↗First posted 30 Mar 2023cs.LGcited by 26

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) allows for the training of competent agents from offline datasets without any interaction with the environment. Online finetuning of such offline models can further improve performance. But how should we ideally finetune agents obtained from offline RL training? While offline RL algorithms can in principle be used for finetuning, in practice, their online performance improves slowly. In contrast, we show that it is possible to use standard online off-policy algorithms for faster improvement. However, we find this approach may suffer from policy collapse, where the policy undergoes severe performance deterioration during initial online learning. We investigate the issue of policy collapse and how it relates to data diversity, algorithm choices and online replay distribution. Based on these insights, we propose a conservative policy optimization procedure that can achieve stable and sample-efficient online learning from offline pretraining.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 20 of its 28 references that are in the map. First posted on arXiv 30 Mar 2023.

Builds on

20 of 28 references.

Closest ancestors · 8

Same direction, newest first

  1. 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation21 Nov
  2. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble1 Jul
  3. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  4. 2020Offline Reinforcement Learning Hands-On29 Nov
  5. 2020Critic Regularized Regression26 Jun
  6. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  7. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  8. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 12

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  4. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  6. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  7. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  8. 2018Maximum a Posteriori Policy Optimisationcited by 318
  9. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  10. 2022Online Decision Transformercited by 194
  11. 2020Acme: A Research Framework for Distributed Reinforcement Learningcited by 146
  12. 2021AW-Opt: Learning Robotic Skills with Imitation and Reinforcement at Scalecited by 43

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 5; 2025: 6; 2026: 7.

26 papers.

Sort:

By area:

  1. 2025Reinforcement Learning with Action Chunkingcited by 97
  2. 2025Blending Supervised and Reinforcement Fine-Tuning with Prefix Samplingcited by 57
  3. 2023PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learningcited by 28
  4. 2023Decision ConvFormer: Local Filtering in MetaFormer is Sufficient for Decision Makingcited by 26
  5. 2023Accelerating Exploration with Unlabeled Prior Datacited by 21
  6. 2025Fast Adaptation with Behavioral Foundation Modelscited by 20
  7. 2024Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RLcited by 9
  8. 2023A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learningcited by 8
  9. 2025Active Advantage-Aligned Online Reinforcement Learning with Offline Datacited by 5
  10. 2026Improving Robotic Generalist Policies via Flow Reversal Steeringcited by 5
  11. 2023Planning to Go Out-of-Distribution in Offline-to-Online Reinforcement Learningcited by 4
  12. 2023Small Dataset, Big Gains: Enhancing Reinforcement Learning by Offline Pre-Training with Model Based Augmentationcited by 4
14 more
  1. 2024Dynamic Normativity: Necessary and Sufficient Conditions for Value Alignmentcited by 3
  2. 2024Offline Behavior Distillationcited by 2
  3. 2025Generative Actor Criticcited by 2
  4. 2023Safe and Interpretable Estimation of Optimal Treatment Regimescited by 1
  5. 2023Improving Offline-to-Online Reinforcement Learning with Q Conditioned State Entropy Explorationcited by 0
  6. 2024Snapshot Reinforcement Learning: Leveraging Prior Trajectories for Efficiencycited by 0
  7. 2024Efficient Offline Reinforcement Learning: First Imitate, then Improvecited by 0
  8. 2025Fine-Tuning without Performance Degradationcited by 0
  9. 2026COOPO: Cyclic Offline-Online Policy Optimization Algorithmcited by 0
  10. 2026Beyond One-Size-Fits-All: Diagnosis-Driven Online Reinforcement Learning with Offline Priorscited by 0
  11. 2026Robo-ValueRL: Reliable Value Estimation for Offline-to-Online Reinforcement Learningcited by 0
  12. 2026Evaluating Fuzz Testing for Reinforcement Learning Agentscited by 0
  13. 2026Schrödinger--Föllmer Actor--Critic: Diffusion Policy Improvement with Finite-Sample Analysiscited by 0
  14. 2026From Static Policies to Adaptive Priors in Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Leveraging Offline Data in Online Reinforcement Learning
  2. 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data
  3. 2026Active Offline-to-Online Reinforcement Learning
  4. 2025The Three Regimes of Offline-to-Online Reinforcement Learning
  5. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning
  6. 2023Offline Retraining for Online RL: Decoupled Policy Learning to Mitigate Exploration Bias
  7. 2023Finetuning Offline World Models in the Real World
  8. 2023Adaptive Policy Learning for Offline-to-Online Reinforcement Learning