mlsanityData to 14 Oct 2026

Iterative Refinement of Flow Policies in Probability Space for Online Reinforcement Learning

arXiv 2510.15388 ↗First posted 17 Oct 2025cs.LGcited by 1

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

While behavior cloning with flow/diffusion policies excels at learning complex skills from demonstrations, it remains vulnerable to distributional shift, and standard RL methods struggle to fine-tune these models due to their iterative inference process and the limitations of existing workarounds. In this work, we introduce the Stepwise Flow Policy (SWFP) framework, founded on the key insight that discretizing the flow matching inference process via a fixed-step Euler scheme inherently aligns it with the variational Jordan-Kinderlehrer-Otto (JKO) principle from optimal transport. SWFP decomposes the global flow into a sequence of small, incremental transformations between proximate distributions. Each step corresponds to a JKO update, regularizing policy changes to stay near the previous iterate and ensuring stable online adaptation with entropic regularization. This decomposition yields an efficient algorithm that fine-tunes pre-trained flows via a cascade of small flow blocks, offering significant advantages: simpler/faster training of sub-models, reduced computational/memory costs, and provable stability grounded in Wasserstein trust regions. Comprehensive experiments demonstrate SWFP's enhanced stability, efficiency, and superior adaptation performance across diverse robotic control benchmarks.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 31 of its 38 references that are in the map. First posted on arXiv 17 Oct 2025.

Builds on

31 of 38 references.

Closest ancestors · 17

Same direction, newest first

  1. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learning18 Jun
  2. 2025Score-Based Diffusion Policy Compatible with Reinforcement Learning via Optimal Transport18 Feb
  3. 2024Diffusion Policy Policy Optimization1 Sep
  4. 2024Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learning29 Jul
  5. 2024Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Review18 Jul
  6. 2024Aligning Diffusion Behaviors with Q-functions for Efficient Continuous Control12 Jul
  7. 2024Riemannian Flow Matching Policy for Robot Motion Learning15 Mar
  8. 20243D Diffuser Actor: Policy Diffusion with 3D Scene Representations16 Feb
  9. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewards29 Sep
  10. 2023Diffusion Policies for Out-of-Distribution Generalization in Offline Reinforcement Learning10 Jul
  11. 2023Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learning29 May
  12. 2023Training Diffusion Models with Reinforcement Learning22 May
5 more
  1. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learning25 Apr
  2. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  3. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar
  4. 2022Is Conditional Generative Modeling all you need for Decision-Making?28 Nov
  5. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 14

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2022Flow Matching for Generative Modelingcited by 5,702
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2018Neural Ordinary Differential Equationscited by 3,779
  5. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,568
  6. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  7. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  8. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  9. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  10. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 237
  11. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 182
  12. 2021Large-Scale Wasserstein Gradient Flowscited by 77
2 more
  1. 2023Imitation Bootstrapped Reinforcement Learningcited by 64
  2. 2022Know Your Boundaries: The Necessity of Explicit Behavioral Cloning in Offline RLcited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026π-StepNFT: Wider Space Needs Finer Steps in Online RL for Flow-based VLAscited by 6

Similar, not cited

Close in wording, no citation either way

  1. 2026Refinement-based Flow Policy Optimization
  2. 2025ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learning
  3. 2026Reinforcement Learning via Fixed-point Flow Matching
  4. 2025FM-IRL: Flow-Matching for Reward Modeling and Policy Regularization in Reinforcement Learning
  5. 2025One-Step Flow Policy Mirror Descent
  6. 2025Flow-Based Policy for Online Reinforcement Learning
  7. 2026Reinforcement Learning for Flow-Matching Policies with Density Transport
  8. 2026A Survey of Flow Matching in Reinforcement Learning