mlsanityData to 7 Oct 2026

SMAC: Score-Matched Actor-Critics for Robust Offline-to-Online Transfer

arXiv 2602.17632 ↗First posted 19 Feb 2026cs.LG cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Modern offline Reinforcement Learning (RL) methods find performant actor-critics, however, fine-tuning these actor-critics online with value-based RL algorithms typically causes immediate drops in performance. We provide evidence consistent with the hypothesis that, in the loss landscape, offline maxima for prior algorithms and online maxima are separated by low-performance valleys that gradient-based fine-tuning traverses. Following this, we present Score Matched Actor-Critic (SMAC), an offline RL method designed to learn actor-critics that transition to online value-based RL algorithms with no drop in performance. SMAC avoids valleys between offline and online maxima by regularizing the Q-function during the offline phase to respect a first-order derivative equality between the score of the policy and action-gradient of the Q-function. We experimentally demonstrate that SMAC converges to offline maxima that are connected to better online maxima via paths with monotonically increasing reward found by first-order optimization. SMAC achieves smooth transfer to Soft Actor-Critic and TD3 in 6/6 D4RL tasks. In 4/6 environments, it reduces regret by 34-58% over the best baseline.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 48 of its 59 references that are in the map. First posted on arXiv 19 Feb 2026.

Builds on

48 of 59 references.

Closest ancestors · 18

Same direction, newest first

  1. 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data10 Dec
  2. 2023A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning12 Dec
  3. 2023Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothness29 Sep
  4. 2023A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learning13 Jun
  5. 2023ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensembles12 Jun
  6. 2023PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learning25 May
  7. 2023Adaptive Policy Learning for Offline-to-Online Reinforcement Learning14 Mar
  8. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning9 Mar
  9. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learning2 Feb
  10. 2021RvS: What is Essential for Offline RL via Supervised Learning?20 Dec
  11. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  12. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble1 Jul
6 more
  1. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  2. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  3. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  4. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  6. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 30

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  4. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  5. 2015Continuous control with deep reinforcement learningcited by 3,279
  6. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  7. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  8. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  9. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  10. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  11. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  12. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
18 more
  1. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  2. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 507
  3. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 504
  4. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 307
  5. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  6. 2016Continuous Deep Q-Learning with Model-based Accelerationcited by 149
  7. 2020Linear Mode Connectivity in Multitask and Continual Learningcited by 144
  8. 2019Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actionscited by 124
  9. 2018An Alternative View: When Does SGD Escape Local Minima?cited by 115
  10. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  11. 2024SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learningcited by 70
  12. 2022Linear Connectivity Reveals Generalization Strategiescited by 55
  13. 2024CleanDiffuser: An Easy-to-use Modularized Library for Diffusion Models in Decision Makingcited by 38
  14. 2021Ensemble Bootstrapping for Q-Learningcited by 30
  15. 2025EXPO: Stable Reinforcement Learning with Expressive Policiescited by 30
  16. 2022The Effect of Modeling Human Rationality Level on Learning Rewards from Multiple Feedback Typescited by 26
  17. 2025What Really Matters in Matrix-Whitening Optimizers?cited by 15
  18. 2023Aligning Robot and Human Representationscited by 11

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL
  2. 2026Simple Actors and Deep Critics for Scalable Reinforcement Learning
  3. 2021You Only Evaluate Once: a Simple Baseline Algorithm for Offline RL
  4. 2026Decoupling Policy Extraction for Offline Reinforcement Learning
  5. 2024Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RL
  6. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning
  7. 2021Offline RL Without Off-Policy Evaluation
  8. 2024Adversarially Trained Weighted Actor-Critic for Safe Offline Reinforcement Learning