mlsanityData to 14 Oct 2026

Offline-to-Online Reinforcement Learning with Classifier-Free Diffusion Generation

arXiv 2508.06806 ↗First posted 9 Aug 2025cs.LG cs.AIcited by 9

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline-to-online Reinforcement Learning (O2O RL) aims to perform online fine-tuning on an offline pre-trained policy to minimize costly online interactions. Existing work used offline datasets to generate data that conform to the online data distribution for data augmentation. However, generated data still exhibits a gap with the online data, limiting overall performance. To address this, we propose a new data augmentation approach, Classifier-Free Diffusion Generation (CFDG). Without introducing additional classifier training overhead, CFDG leverages classifier-free guidance diffusion to significantly enhance the generation quality of offline and online data with different distributions. Additionally, it employs a reweighting method to enable more generated data to align with the online data, enhancing performance while maintaining the agent's stability. Experimental results show that CFDG outperforms replaying the two data types or using a standard diffusion model to generate new data. Our method is versatile and can be integrated with existing offline-to-online RL algorithms. By implementing CFDG to popular methods IQL, PEX and APL, we achieve a notable 15% average improvement in empirical performance on the D4RL benchmark such as MuJoCo and AntMaze.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 28 of its 31 references that are in the map. First posted on arXiv 9 Aug 2025.

Builds on

28 of 31 references.

Closest ancestors · 13

Same direction, newest first

  1. 2023A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learning13 Jun
  2. 2023PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learning25 May
  3. 2023Adaptive Policy Learning for Offline-to-Online Reinforcement Learning14 Mar
  4. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning9 Mar
  5. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learning2 Feb
  6. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  7. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble1 Jul
  8. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  9. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  10. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  11. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  12. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
1 more
  1. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 15

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  3. 2022Classifier-Free Diffusion Guidancecited by 5,847
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  5. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  6. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  7. 2016Variational Inference: A Review for Statisticianscited by 1,472
  8. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  9. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  10. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  11. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  12. 2023Imitating Human Behaviour with Diffusion Modelscited by 263
3 more
  1. 2023Synthetic Experience Replaycited by 101
  2. 2024Energy-Guided Diffusion Sampling for Offline-to-Online Reinforcement Learningcited by 17
  3. 2017Enhanced Experience Replay Generation for Efficient Reinforcement Learningcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 9.

9 papers.

  1. 2026Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learningcited by 11
  2. 2026FlowRL: A Taxonomy and Modular Framework for Reinforcement Learning with Diffusion Policiescited by 3
  3. 2026RankQ: Offline-to-Online Reinforcement Learning via Self-Supervised Action Rankingcited by 2
  4. 2026Flow Matching with Injected Noise for Offline-to-Online Reinforcement Learningcited by 1
  5. 2026Provably Efficient Offline-to-Online Value Adaptation with General Function Approximationcited by 0
  6. 2026ROAD: Adaptive Data Mixing for Offline-to-Online Reinforcement Learning via Bi-Level Optimizationcited by 0
  7. 2026Efficient and Uncertainty-Aware Diffusion Framework for Offline-to-Online Reinforcement Learningcited by 0
  8. 2026Diffusion Offline Reinforcement Learning for Fair and Energy-Efficient UAV-Assisted Wireless Networkscited by 0
  9. 2026Optimal Transport Q-Learning for Flow Policy Steering and Accelerationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Critic-Free Pretraining for Efficient Online Reinforcement Learning Fine-Tuning
  2. 2025Efficient Reinforcement Learning by Guiding World Models with Non-Curated Data
  3. 2026Active Offline-to-Online Reinforcement Learning
  4. 2024SAMG: Offline-to-Online Reinforcement Learning via State-Action-Conditional Offline Model Guidance
  5. 2021Online and Offline Reinforcement Learning by Planning with a Learned Model
  6. 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data
  7. 2023Planning to Go Out-of-Distribution in Offline-to-Online Reinforcement Learning
  8. 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments