mlsanityData to 7 Oct 2026

Flow Matching with Injected Noise for Offline-to-Online Reinforcement Learning

arXiv 2602.18117 ↗First posted 20 Feb 2026cs.LG cs.AIcited by 0

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Generative models have recently demonstrated remarkable success across diverse domains, motivating their adoption as expressive policies in reinforcement learning (RL). While they have shown strong performance in offline RL, particularly where the target distribution is well defined, their extension to online fine-tuning has largely been treated as a direct continuation of offline pre-training, leaving key challenges unaddressed. In this paper, we propose Flow Matching with Injected Noise for Offline-to-Online RL (FINO), a novel method that leverages flow matching-based policies to enhance sample efficiency for offline-to-online RL. FINO facilitates effective exploration by injecting noise into policy training, thereby encouraging a broader range of actions beyond those observed in the offline dataset. In addition to exploration-enhanced flow policy training, we combine an entropy-guided sampling mechanism to balance exploration and exploitation, allowing the policy to adapt its behavior throughout online fine-tuning. Experiments across diverse, challenging tasks demonstrate that FINO consistently achieves superior performance under limited online budgets.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 39 of its 49 references that are in the map. First posted on arXiv 20 Feb 2026.

Builds on

39 of 49 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025ImageReFL: Balancing Quality and Diversity in Human-Aligned Diffusion Models28 May
  2. 2025Energy-Weighted Flow Matching for Offline Reinforcement Learning6 Mar
  3. 2025Online Reward-Weighted Fine-Tuning of Flow Matching with Wasserstein Regularization9 Feb
  4. 2025Flow Q-Learning4 Feb
  5. 2024Energy-Guided Diffusion Sampling for Offline-to-Online Reinforcement Learning17 Jul
  6. 2024Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning31 May
  7. 2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization25 May
  8. 2024Diffusion Actor-Critic with Entropy Regulator24 May
  9. 2023Learning a Diffusion Model Policy from Rewards via Q-Score Matching18 Dec
  10. 2023Efficient Diffusion Policies for Offline Reinforcement Learning31 May
  11. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  12. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 27

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  5. 2022Flow Matching for Generative Modelingcited by 5,426
  6. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  8. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  9. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  10. 2022Building Normalizing Flows with Stochastic Interpolantscited by 964
  11. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  12. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
15 more
  1. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 216
  2. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 211
  3. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
  4. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 103
  5. 2024Normalizing Flows are Capable Generative Modelscited by 89
  6. 2022Supported Policy Optimization for Offline Reinforcement Learningcited by 73
  7. 2024Preserving Diversity in Supervised Fine-Tuning of Large Language Modelscited by 68
  8. 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Datacited by 58
  9. 2023Decision ConvFormer: Local Filtering in MetaFormer is Sufficient for Decision Makingcited by 26
  10. 2023Train Once, Get a Family: State-Adaptive Balances for Offline-to-Online Reinforcement Learningcited by 26
  11. 2023A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learningcited by 15
  12. 2024Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RLcited by 9
  13. 2025Offline-to-Online Reinforcement Learning with Classifier-Free Diffusion Generationcited by 9
  14. 2024Adaptive Q-Aid for Conditional Supervised Learning in Offline Reinforcement Learningcited by 7
  15. 2025Penalizing Infeasible Actions and Reward Scaling in Reinforcement Learning with Offline Datacited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025EXPO: Stable Reinforcement Learning with Expressive Policies
  2. 2025Composite Flow Matching for Reinforcement Learning with Shifted-Dynamics Data
  3. 2026Flow Matching Policy Optimization with Mirror Descent and Entropy Constraints
  4. 2026Discrete Flow Matching for Offline-to-Online Reinforcement Learning
  5. 2026Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learning
  6. 2025Robust Policy Expansion for Offline-to-Online RL under Diverse Data Corruption
  7. 2026Score-Calibrated Flow for Sampling from Unnormalized Densities with Applications to Generative Online Reinforcement Learning
  8. 2026Reverse Flow Matching: A Unified Framework for Online Reinforcement Learning with Diffusion and Flow Policies