mlsanityData to 7 Oct 2026

SAC Flow: Sample-Efficient Reinforcement Learning of Flow-Based Policies via Velocity-Reparameterized Sequential Modeling

arXiv 2509.25756 ↗First posted 30 Sep 2025cs.RO cs.LGcited by 27

Not on the map. Why?

Abstract (arXiv)

Training expressive flow-based policies with off-policy reinforcement learning is notoriously unstable due to gradient pathologies in the multi-step action sampling process. We trace this instability to a fundamental connection: the flow rollout is algebraically equivalent to a residual recurrent computation, making it susceptible to the same vanishing and exploding gradients as RNNs. To address this, we reparameterize the velocity network using principles from modern sequential models, introducing two stable architectures: Flow-G, which incorporates a gated velocity, and Flow-T, which utilizes a decoded velocity. We then develop a practical SAC-based algorithm, enabled by a noise-augmented rollout, that facilitates direct end-to-end training of these policies. Our approach supports both from-scratch and offline-to-online learning and achieves state-of-the-art performance on continuous control and robotic manipulation benchmarks, eliminating the need for common workarounds like policy distillation or surrogate objectives.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 32 of its 36 references that are in the map. First posted on arXiv 30 Sep 2025.

Builds on

32 of 36 references.

References · 32

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2022Flow Matching for Generative Modelingcited by 5,426
  3. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  4. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  5. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  6. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  7. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  8. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  9. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  10. 2024Diffusion Policy Policy Optimizationcited by 255
  11. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 216
  12. 2025Flow Q-Learningcited by 162
20 more
  1. 2023A Survey of Imitation Learning: Algorithms, Recent Developments, and Challengescited by 145
  2. 2023Learning a Diffusion Model Policy from Rewards via Q-Score Matchingcited by 126
  3. 2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimizationcited by 99
  4. 2025Reinforcement Learning with Action Chunkingcited by 97
  5. 2025Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videoscited by 94
  6. 2025ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learningcited by 83
  7. 2024From Imitation to Refinement -- Residual RL for Precise Assemblycited by 80
  8. 2023Data Quality in Imitation Learningcited by 72
  9. 2024Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbonecited by 71
  10. 2025DIME:Diffusion-Based Maximum Entropy Reinforcement Learningcited by 52
  11. 2024Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradientcited by 49
  12. 2024Towards Generalizable Vision-Language Robotic Manipulation: A Benchmark and LLM-guided 3D Policycited by 37
  13. 2025Streaming Flow Policy: Simplifying diffusion/flow-matching policies by treating action trajectories as flow trajectoriescited by 34
  14. 2025Flow-Based Policy for Online Reinforcement Learningcited by 33
  15. 2025Maximum Entropy Reinforcement Learning with Diffusion Policycited by 29
  16. 2017R2N2: Residual Recurrent Neural Networks for Multivariate Time Series Forecastingcited by 10
  17. 2025D3P: Dynamic Denoising Diffusion Policy via Reinforcement Learningcited by 10
  18. 2023Wasserstein Gradient Flows for Optimizing Gaussian Mixture Policiescited by 8
  19. 2025NinA: Normalizing Flows in Action. Training VLA Models with Normalizing Flowscited by 5
  20. 2024When Do Off-Policy and On-Policy Policy Gradient Methods Align?cited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 26.

27 papers.

Sort:

By area:

  1. 2026Beyond Imitation: Reinforcement Learning-Based Sim-Real Co-Training for VLA Modelscited by 8
  2. 2026ALOE: Action-Level Off-Policy Evaluation for Vision-Language-Action Model Post-Trainingcited by 7
  3. 2026DeFlow: Decoupling Manifold Modeling and Value Maximization for Offline Policy Extractioncited by 5
  4. 2026FLAC: Maximum Entropy RL via Kinetic Energy Regularized Bridge Matchingcited by 5
  5. 2026SEAR: Sample Efficient Action Chunking Reinforcement Learningcited by 5
  6. 2026Reparameterization Flow Policy Optimizationcited by 4
  7. 2026Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matchingcited by 4
  8. 2025Training Diffusion Policies via Prior-Mapping Co-Evolutioncited by 3
  9. 2026Trust Region Q Adjoint Matchingcited by 3
  10. 2026Truncated Rectified Flow Policy for Reinforcement Learning with One-Step Samplingcited by 2
  11. 2026Reversal Q-Learningcited by 2
  12. 2026Causal Flow Q-Learning for Robust Offline Reinforcement Learningcited by 1
15 more
  1. 2026GeMPO: Generalized Measure Matching for Online Diffusion Reinforcement Learningcited by 1
  2. 2026Flow Matching Policy Optimization with Mirror Descent and Entropy Constraintscited by 1
  3. 2026AcceRL: A Distributed Asynchronous Reinforcement Learning and World Model Framework for Vision-Language-Action Modelscited by 1
  4. 2026Mean Flow Policy Optimizationcited by 1
  5. 2026Decentralized Diffusion Policy Learning for Enhanced Exploration in Cooperative Multi-agent Reinforcement Learningcited by 1
  6. 2026Generative Actor-Critic with Soft Bridge Policiescited by 1
  7. 2026Dual-Flow Reinforcement Learning with State-Aware Explorationcited by 1
  8. 2026Path-Space Mirror Descent for On-Policy Reinforcement Learning under the Generalized Schr"odinger Bridgecited by 0
  9. 2026Flow-based Policy With Distributional Reinforcement Learning in Trajectory Optimizationcited by 0
  10. 2026Stochastic MeanFlow Policies: One-Step Generative Control with Entropic Mirror Descentcited by 0
  11. 2026Reinforcement Learning for Flow-Matching Policies with Density Transportcited by 0
  12. 2026VINE: Taming Generative Control Policies for Reinforcement Learningcited by 0
  13. 2026Critic-Free Pretraining for Efficient Online Reinforcement Learning Fine-Tuningcited by 0
  14. 2026Safe Score Matching: Diffusion Policies with Hamilton-Jacobi Reachability for Online Safe Reinforcement Learningcited by 0
  15. 2026Flowing Faster to Coordinate: One-Step Online Multi-Agent Flow Policiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026RoMAN-Flow: Taming Autoregressive Normalizing Flows for Offline Reinforcement Learning in Robotic Manipulation
  2. 2026FlowDPG: Deterministic Policy Gradient on Flow Matching Policies for Real-World Manipulation
  3. 2026RFPO: Rectified Flow Policy Optimization for Embodied Control
  4. 2026Refinement-based Flow Policy Optimization
  5. 2026VGFM: Expressive Robot Policies via Dense Value Guidance in Flow Matching
  6. 2025Guided Flow Policy: Learning from High-Value Actions in Offline Reinforcement Learning
  7. 2025FlowQ: Energy-Guided Flow Policies for Offline Reinforcement Learning
  8. 2025Controllable Flow Matching for Online Reinforcement Learning