mlsanityData to 7 Oct 2026

Fine-Tuning without Performance Degradation

arXiv 2505.00913 ↗First posted 1 May 2025cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Fine-tuning policies learned offline remains a major challenge in application domains. Monotonic performance improvement during fine-tuning is often challenging, as agents typically experience performance degradation at the early fine-tuning stage. The community has identified multiple difficulties in fine-tuning a learned network online, however, the majority of progress has focused on improving learning efficiency during fine-tuning. In practice, this comes at a serious cost during fine-tuning: initially, agent performance degrades as the agent explores and effectively overrides the policy learned offline. We show across a range of settings, many offline-to-online algorithms exhibit either (1) performance degradation or (2) slow learning (sometimes effectively no improvement) during fine-tuning. We introduce a new fine-tuning algorithm, based on an algorithm called Jump Start, that gradually allows more exploration based on online estimates of performance. Empirically, this approach achieves fast fine-tuning and significantly reduces performance degradations compared with existing algorithms designed to do the same.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 29 of its 32 references that are in the map. First posted on arXiv 1 May 2025.

Builds on

29 of 32 references.

References · 29

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  4. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  5. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  6. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  7. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  8. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 297
  9. 2019Batch Policy Learning under Constraintscited by 242
  10. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  11. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  12. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 211
17 more
  1. 2019To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Taskscited by 167
  2. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
  3. 2020Better Fine-Tuning by Reducing Representational Collapsecited by 156
  4. 2020What Happens To BERT Embeddings During Fine-tuning?cited by 113
  5. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
  6. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 103
  7. 2022Jump-Start Reinforcement Learningcited by 102
  8. 2021Legged Robots that Keep on Learning: Fine-Tuning Locomotion Policies in the Real Worldcited by 64
  9. 2017Lifelong Generative Modelingcited by 51
  10. 2021A Closer Look at How Fine-tuning Changes BERTcited by 42
  11. 2016Unifying task specification in reinforcement learningcited by 41
  12. 2022Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learningcited by 34
  13. 2023The In-Sample Softmax for Offline Reinforcement Learningcited by 33
  14. 2023PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learningcited by 28
  15. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutionscited by 26
  16. 2021Beyond Fine-Tuning: Transferring Behavior in Reinforcement Learningcited by 21
  17. 2023ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensemblescited by 20

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data
  2. 2026Critic-Free Pretraining for Efficient Online Reinforcement Learning Fine-Tuning
  3. 2026Online Self-Weighted Fine-Tuning
  4. 2024Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RL
  5. 2025The Three Regimes of Offline-to-Online Reinforcement Learning
  6. 2025Adversarial Fine-tuning in Offline-to-Online Reinforcement Learning for Robust Robot Control
  7. 2026Active Offline-to-Online Reinforcement Learning
  8. 2025Efficient Online RL Fine Tuning with Offline Pre-trained Policy Only