mlsanityData to 7 Oct 2026

One-Step Flow Policy Mirror Descent

arXiv 2507.23675 ↗First posted 31 Jul 2025cs.LGcited by 18

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

Diffusion policies have achieved great success in online reinforcement learning (RL) due to their strong expressive capacity. However, the inference of diffusion policy models relies on a slow iterative sampling process, which limits their responsiveness. To overcome this limitation, we propose Flow Policy Mirror Descent (FPMD), an online RL algorithm that enables 1-step sampling during flow policy inference. Our approach exploits a theoretical connection between the distribution variance and the discretization error of single-step sampling in straight interpolation flow matching models, and requires no extra distillation or consistency training. We present two algorithm variants based on rectified flow policy and MeanFlow policy, respectively. Extensive empirical evaluations on MuJoCo and visual DeepMind Control Suite benchmarks demonstrate that our algorithms show strong performance comparable to diffusion policy baselines while requiring orders of magnitude less computational cost during inference.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 60 of its 68 references that are in the map. First posted on arXiv 31 Jul 2025.

Builds on

60 of 68 references.

Closest ancestors · 21

Same direction, newest first

  1. 2025Flow Matching Policy Gradients28 Jul
  2. 2025Reinforcement Learning for Flow-Matching Policies20 Jul
  3. 2025Flow-Based Single-Step Completion for Efficient and Expressive Policy Learning26 Jun
  4. 2025ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learning28 May
  5. 2025DanceGRPO: Unleashing GRPO on Visual Generation12 May
  6. 2025Flow-GRPO: Training Flow Matching Models via Online RL8 May
  7. 2025Energy-Weighted Flow Matching for Offline Reinforcement Learning6 Mar
  8. 2025Online Reward-Weighted Fine-Tuning of Flow Matching with Wasserstein Regularization9 Feb
  9. 2025Flow Q-Learning4 Feb
  10. 2025DIME:Diffusion-Based Maximum Entropy Reinforcement Learning4 Feb
  11. 2025Efficient Online Reinforcement Learning for Diffusion Policy1 Feb
  12. 2024Diffusion Policy Policy Optimization1 Sep
9 more
  1. 2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization25 May
  2. 2024Diffusion Actor-Critic with Entropy Regulator24 May
  3. 2024Consistency Policy: Accelerated Visuomotor Policies via Consistency Distillation13 May
  4. 20243D Diffuser Actor: Policy Diffusion with 3D Scene Representations16 Feb
  5. 2023Learning a Diffusion Model Policy from Rewards via Q-Score Matching18 Dec
  6. 2023Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learning29 Sep
  7. 2023Policy Representation via Diffusion Probability Model for Reinforcement Learning22 May
  8. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar
  9. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  6. 2022Flow Matching for Generative Modelingcited by 5,426
  7. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  8. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  9. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  10. 2015Continuous control with deep reinforcement learningcited by 3,279
  11. 2015Trust Region Policy Optimizationcited by 2,503
  12. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
27 more
  1. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  2. 2023Mastering Diverse Domains through World Modelscited by 1,193
  3. 2022Building Normalizing Flows with Stochastic Interpolantscited by 964
  4. 2018DeepMind Control Suitecited by 935
  5. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  6. 2025Mean Flows for One-step Generative Modelingcited by 524
  7. 2023Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusioncited by 448
  8. 2023InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generationcited by 387
  9. 2024One Step Diffusion via Shortcut Modelscited by 373
  10. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  11. 2024Discrete Flow Matchingcited by 301
  12. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
  13. 2023Multisample Flow Matching: Straightening Flows with Minibatch Couplingscited by 267
  14. 2018Distributed Distributional Deterministic Policy Gradientscited by 195
  15. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 177
  16. 2021Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classescited by 149
  17. 2024Improving the Training of Rectified Flowscited by 106
  18. 2020Mirror Descent Policy Optimizationcited by 102
  19. 2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 82
  20. 2024Optimal Flow Matching: Learning Straight Trajectories in Just One Stepcited by 74
  21. 2024AdaFlow: Imitation Learning with Variance-Adaptive Flow-Based Policiescited by 50
  22. 2023Movement Primitive Diffusion: Learning Gentle Robotic Manipulation of Deformable Objectscited by 40
  23. 2025Flow-Based Policy for Online Reinforcement Learningcited by 33
  24. 2025MP1: MeanFlow Tames Policy Learning in 1-step for Robotic Manipulationcited by 32
  25. 2020Batch Stationary Distribution Estimationcited by 19
  26. 2023Optimal Convergence Rate for Exact Policy Mirror Descent in Discounted Markov Decision Processescited by 17
  27. 2023Bellman Optimal Stepsize Straightening of Flow-Matching Modelscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 17.

18 papers.

Sort:

By area:

  1. 2026Q-learning with Adjoint Matchingcited by 33
  2. 2026Test-Time Gradient Guidance of Flow Policies in Reinforcement Learningcited by 5
  3. 2026Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matchingcited by 4
  4. 2025Training Diffusion Policies via Prior-Mapping Co-Evolutioncited by 3
  5. 2026GoldenStart: Q-Guided Priors and Entropy Control for Distilling Flow Policiescited by 3
  6. 2026FlowRL: A Taxonomy and Modular Framework for Reinforcement Learning with Diffusion Policiescited by 2
  7. 2026Truncated Rectified Flow Policy for Reinforcement Learning with One-Step Samplingcited by 2
  8. 2026FLAG: Flow Policy MaxEnt-RL by Latent Augmented Guidancecited by 2
  9. 2026Advances in GRPO for Generation Models: A Surveycited by 1
  10. 2026Mean Flow Policy Optimizationcited by 1
  11. 2026Discrete MeanFlow: One-Step Generation via Conditional Transition Kernelscited by 0
  12. 2026Stochastic MeanFlow Policies: One-Step Generative Control with Entropic Mirror Descentcited by 0
6 more
  1. 2026Reinforcement Learning for Flow-Matching Policies with Density Transportcited by 0
  2. 2026ReFPO: Reflow Regularization for Flow Matching Policy Gradientscited by 0
  3. 2026VINE: Taming Generative Control Policies for Reinforcement Learningcited by 0
  4. 2026G2MAF: Test-Time Gradient Guidance for Multi-Agent Flow Policiescited by 0
  5. 2026QAMM: Adjoint MeanFlow Matching for Few-Step Offline Reinforcement Learningcited by 0
  6. 2026Flowing Faster to Coordinate: One-Step Online Multi-Agent Flow Policiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Flow Matching Policy Optimization with Mirror Descent and Entropy Constraints
  2. 2026Reverse Flow Matching: A Unified Framework for Online Reinforcement Learning with Diffusion and Flow Policies
  3. 2025DiffusionNFT: Online Diffusion Reinforcement with Forward Process
  4. 2025Iterative Refinement of Flow Policies in Probability Space for Online Reinforcement Learning
  5. 2025Reinforcement Learning with Discrete Diffusion Policies for Combinatorial Action Spaces
  6. 2026Score-Based One-step MeanFlow Policy Optimization
  7. 2025One-Step Flow Q-Learning: Addressing the Diffusion Policy Bottleneck in Offline Reinforcement Learning
  8. 2023Score Regularized Policy Optimization through Diffusion Behavior