# Flow Matching with Injected Noise for Offline-to-Online Reinforcement Learning

arXiv: https://arxiv.org/abs/2602.18117 · HTML: https://mlsanity.com/paper/2602.18117/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models.
- Builds on 39 of its 49 references that are in the map.
- First posted on arXiv 20 Feb 2026.

## Placed in

- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)
- Offline RL and imitation learning from data / [Diffusion and flow policies for offline RL](https://mlsanity.com/offlinerl/diffusion-and-flow-policies-for-offline-rl/)

## Builds on (most cited first, 30 of 39)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [Building Normalizing Flows with Stochastic Interpolants](https://mlsanity.com/paper/2209.15571/) (2022)
- [Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning](https://mlsanity.com/paper/2208.06193/) (2022)
- [Efficient Online Reinforcement Learning with Offline Data](https://mlsanity.com/paper/2302.02948/) (2023)
- [IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies](https://mlsanity.com/paper/2304.10573/) (2023)
- [OGBench: Benchmarking Offline Goal-Conditioned RL](https://mlsanity.com/paper/2410.20092/) (2024)
- [Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning](https://mlsanity.com/paper/2303.05479/) (2023)
- [Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble](https://mlsanity.com/paper/2107.00591/) (2021)
- [Flow Q-Learning](https://mlsanity.com/paper/2502.02538/) (2025)
- [Efficient Diffusion Policies for Offline Reinforcement Learning](https://mlsanity.com/paper/2305.20081/) (2023)
- [Learning a Diffusion Model Policy from Rewards via Q-Score Matching](https://mlsanity.com/paper/2312.11752/) (2023)
- [Mildly Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2206.04745/) (2022)
- [Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization](https://mlsanity.com/paper/2405.16173/) (2024)
- [Policy Expansion for Bridging Offline-to-Online Reinforcement Learning](https://mlsanity.com/paper/2302.00935/) (2023)
- [Diffusion Actor-Critic with Entropy Regulator](https://mlsanity.com/paper/2405.15177/) (2024)
- [Normalizing Flows are Capable Generative Models](https://mlsanity.com/paper/2412.06329/) (2024)
- [Supported Policy Optimization for Offline Reinforcement Learning](https://mlsanity.com/paper/2202.06239/) (2022)
- [Energy-Weighted Flow Matching for Offline Reinforcement Learning](https://mlsanity.com/paper/2503.04975/) (2025)
- [Preserving Diversity in Supervised Fine-Tuning of Large Language Models](https://mlsanity.com/paper/2408.16673/) (2024)
- [Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data](https://mlsanity.com/paper/2412.07762/) (2024)
- [Online Reward-Weighted Fine-Tuning of Flow Matching with Wasserstein Regularization](https://mlsanity.com/paper/2502.06061/) (2025)
- [Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning](https://mlsanity.com/paper/2405.20555/) (2024)

## Built on by (most cited first, 1 of 1)

- [Reinforcement Learning via Fixed-point Flow Matching](https://mlsanity.com/paper/2610.12848/) (2026)

## Abstract (arXiv)

Generative models have recently demonstrated remarkable success across diverse domains, motivating their adoption as expressive policies in reinforcement learning (RL). While they have shown strong performance in offline RL, particularly where the target distribution is well defined, their extension to online fine-tuning has largely been treated as a direct continuation of offline pre-training, leaving key challenges unaddressed. In this paper, we propose Flow Matching with Injected Noise for Offline-to-Online RL (FINO), a novel method that leverages flow matching-based policies to enhance sample efficiency for offline-to-online RL. FINO facilitates effective exploration by injecting noise into policy training, thereby encouraging a broader range of actions beyond those observed in the offline dataset. In addition to exploration-enhanced flow policy training, we combine an entropy-guided sampling mechanism to balance exploration and exploitation, allowing the policy to adapt its behavior throughout online fine-tuning. Experiments across diverse, challenging tasks demonstrate that FINO consistently achieves superior performance under limited online budgets.