mlsanityData to 14 Oct 2026

SAMG: Offline-to-Online Reinforcement Learning via State-Action-Conditional Offline Model Guidance

arXiv 2410.18626 ↗First posted 24 Oct 2024cs.LG cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline-to-online (O2O) reinforcement learning (RL) pre-trains models on offline data and refines policies through online fine-tuning. However, existing O2O RL algorithms typically require maintaining the tedious offline datasets to mitigate the effects of out-of-distribution (OOD) data, which significantly limits their efficiency in exploiting online samples. To address this deficiency, we introduce a new paradigm for O2O RL called State-Action-Conditional Offline Model Guidance (SAMG). It freezes the pre-trained offline critic to provide compact offline understanding for each state-action sample, thus eliminating the need for retraining on offline data. The frozen offline critic is incorporated with the online target critic weighted by a state-action-adaptive coefficient. This coefficient aims to capture the offline degree of samples at the state-action level, and is updated adaptively during training. In practice, SAMG could be easily integrated with Q-function-based algorithms. Theoretical analysis shows good optimality and lower estimation error. Empirically, SAMG outperforms state-of-the-art O2O RL algorithms on the D4RL benchmark.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 46 of its 62 references that are in the map. First posted on arXiv 24 Oct 2024.

Builds on

46 of 62 references.

Closest ancestors · 20

Same direction, newest first

  1. 2023Offline Data Enhanced On-Policy Policy Gradient with Provable Guarantees14 Nov
  2. 2023Train Once, Get a Family: State-Adaptive Balances for Offline-to-Online Reinforcement Learning27 Oct
  3. 2023Planning to Go Out-of-Distribution in Offline-to-Online Reinforcement Learning9 Oct
  4. 2023Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothness29 Sep
  5. 2023A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learning13 Jun
  6. 2023ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensembles12 Jun
  7. 2023Adaptive Policy Learning for Offline-to-Online Reinforcement Learning14 Mar
  8. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning9 Mar
  9. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learning2 Feb
  10. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Library13 Oct
  11. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
  12. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
8 more
  1. 2022Supported Policy Optimization for Offline Reinforcement Learning13 Feb
  2. 2022MOORe: Model-based Offline-to-Online Reinforcement Learning25 Jan
  3. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  4. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble1 Jul
  5. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  6. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  7. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  8. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 26

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  3. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  4. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  5. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  6. 2015Generating Sentences from a Continuous Spacecited by 612
  7. 2014Semi-Supervised Learning with Deep Generative Modelscited by 496
  8. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  9. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 345
  10. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 340
  11. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 319
  12. 2020Never Give Up: Learning Directed Exploration Strategiescited by 212
14 more
  1. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 161
  2. 2018Reinforcement and Imitation Learning for Diverse Visuomotor Skillscited by 126
  3. 2018Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Controlcited by 113
  4. 2018Playing hard exploration games by watching YouTubecited by 99
  5. 2023STEVE-1: A Generative Model for Text-to-Behavior in Minecraftcited by 97
  6. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 91
  7. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learningcited by 91
  8. 2018Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Costcited by 84
  9. 2022A Review of Deep Transfer Learning and Recent Advancementscited by 57
  10. 2023Posterior Collapse and Latent Variable Non-identifiabilitycited by 57
  11. 2022Redeeming Intrinsic Rewards via Constrained Optimizationcited by 24
  12. 2022Improving TD3-BC: Relaxed Policy Constraint for Offline Learning and Stable Online Fine-Tuningcited by 20
  13. 2024Energy-Guided Diffusion Sampling for Offline-to-Online Reinforcement Learningcited by 17
  14. 2024OLLIE: Imitation Learning from Offline Pretraining to Online Finetuningcited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2020MOPO: Model-based Offline Policy Optimization
  2. 2024Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RL
  3. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning
  4. 2026Active Offline-to-Online Reinforcement Learning
  5. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions
  6. 2025A Clean Slate for Offline Reinforcement Learning
  7. 2024Preferred-Action-Optimized Diffusion Policies for Offline Reinforcement Learning
  8. 2021Online and Offline Reinforcement Learning by Planning with a Learned Model