mlsanityData to 14 Oct 2026

Critic-Free Pretraining for Efficient Online Reinforcement Learning Fine-Tuning

arXiv 2608.10473 ↗First posted 11 Aug 2026cs.LG cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

also in 1 more
Abstract (arXiv)

Offline-to-online (O2O) reinforcement learning aims to leverage policies pretrained on static datasets while improving them through online interaction. However, directly reusing an offline-trained critic can hinder online fine-tuning: as the policy and data distribution change rapidly, value estimates inherited from offline training may become misaligned with the online environment, leading to inaccurate policy improvement and inefficient exploration. To address this problem, we introduce Critic-Free Pretraining: an efficient paradigm that completely abandons the approach of offline critic training, allowing a freshly initialized critic to adapt without inheriting biased estimates. CFP is compatible with various mainstream O2O algorithms and consistently matches or improves upon conventional O2O algorithms across a diverse set of tasks, with particularly pronounced gains on several challenging tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 35 of its 43 references that are in the map. First posted on arXiv 11 Aug 2026.

Builds on

35 of 43 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Reinforcement Learning with Action Chunking10 Jul
  2. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning9 Mar
  3. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learning2 Feb
  4. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  5. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  6. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning9 Jun
  7. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
  8. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  9. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May

Other references · 26

  1. 2022Flow Matching for Generative Modelingcited by 5,702
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  4. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  6. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  7. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 725
  8. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  9. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 562
  10. 2018Behavioral Cloning from Observationcited by 492
  11. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  12. 2017Deep Reinforcement Learning: An Overviewcited by 333
14 more
  1. 2024Flow Matching Guide and Codecited by 301
  2. 2024Diffusion Policy Policy Optimizationcited by 280
  3. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 238
  4. 2025Flow Q-Learningcited by 184
  5. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 169
  6. 2023Efficient Diffusion Policies for Offline Reinforcement Learningcited by 157
  7. 2025Flow Matching Policy Gradientscited by 99
  8. 2025ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learningcited by 97
  9. 2024Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learningcited by 74
  10. 2026Q-learning with Adjoint Matchingcited by 37
  11. 2025SAC Flow: Sample-Efficient Reinforcement Learning of Flow-Based Policies via Velocity-Reparameterized Sequential Modelingcited by 31
  12. 2026Q-Flow: Stable and Expressive Reinforcement Learning with Flow-Based Policycited by 6
  13. 2026PolicyFlow: Policy Optimization with Continuous Normalizing Flow in Reinforcement Learningcited by 5
  14. 2026Rainbow-DemoRL: Combining Improvements in Demonstration-Augmented Reinforcement Learningcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RL
  2. 2026Active Offline-to-Online Reinforcement Learning
  3. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions
  4. 2025Offline-to-Online Reinforcement Learning with Classifier-Free Diffusion Generation
  5. 2023Planning to Go Out-of-Distribution in Offline-to-Online Reinforcement Learning
  6. 2025Online Pre-Training for Offline-to-Online Reinforcement Learning
  7. 2026COOPO: Cyclic Offline-Online Policy Optimization Algorithm
  8. 2024SAMG: Offline-to-Online Reinforcement Learning via State-Action-Conditional Offline Model Guidance