mlsanityData to 14 Oct 2026

Stochastic MeanFlow Policies: One-Step Generative Control with Entropic Mirror Descent

arXiv 2605.21282 ↗First posted 20 May 2026cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → RL with diffusion and flow models

Abstract (arXiv)

Combining entropy regularisation with mirror descent in online off-policy reinforcement learning requires expressive policies with efficient sampling and explicit entropy control. Gaussian policies struggle to represent multimodal targets, whereas generative policies often require iterative sampling or lack tractable entropy estimates. We introduce Stochastic MeanFlow Policies (SMFP), a one-step generative policy class that combines MeanFlow-based noise-to-action mappings with Gaussian reparameterisation. The latent mapping captures diverse action modes, while a learnable conditional Gaussian scale provides an analytic entropy lower bound. We use this bound to regulate stochasticity alongside direct value guidance and advantage-weighted MeanFlow regression, enabling entropy-regularised mirror-descent learning with one-step policies. Across seven MuJoCo benchmarks, SMFP achieves strong performance against Gaussian and generative baselines with single-step inference. Additional experiments demonstrate multimodal action modelling and effective exploration in sparse-reward navigation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL with diffusion and flow models, one of 15 directions in Deep reinforcement learning methods. Builds on 48 of its 59 references that are in the map. First posted on arXiv 20 May 2026.

Builds on

48 of 59 references.

Closest ancestors · 12

Same direction, newest first

  1. 2026FLAC: Maximum Entropy RL via Kinetic Energy Regularized Bridge Matching13 Feb
  2. 2025One-Step Generative Policies with Q-Learning: A Reformulation of MeanFlow17 Nov
  3. 2025SAC Flow: Sample-Efficient Reinforcement Learning of Flow-Based Policies via Velocity-Reparameterized Sequential Modeling30 Sep
  4. 2025One-Step Flow Policy Mirror Descent31 Jul
  5. 2025Flow-Based Policy for Online Reinforcement Learning15 Jun
  6. 2025ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learning28 May
  7. 2025Maximum Entropy Reinforcement Learning with Diffusion Policy17 Feb
  8. 2025Flow Q-Learning4 Feb
  9. 2025DIME:Diffusion-Based Maximum Entropy Reinforcement Learning4 Feb
  10. 2025Efficient Online Reinforcement Learning for Diffusion Policy1 Feb
  11. 2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization25 May
  12. 2024Diffusion Actor-Critic with Entropy Regulator24 May

Other references · 36

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  4. 2022Scalable Diffusion Models with Transformerscited by 6,054
  5. 2022Flow Matching for Generative Modelingcited by 5,702
  6. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  7. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  8. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  9. 2015Trust Region Policy Optimizationcited by 2,580
  10. 2024Octo: An Open-Source Generalist Robot Policycited by 1,484
  11. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  12. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
24 more
  1. 2018FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Modelscited by 594
  2. 2025Mean Flows for One-step Generative Modelingcited by 561
  3. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  4. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 353
  5. 2018Maximum a Posteriori Policy Optimisationcited by 328
  6. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 325
  7. 2017A unified view of entropy-regularized Markov decision processescited by 182
  8. 2024Deep Reinforcement Learning for Robotics: A Survey of Real-World Successescited by 176
  9. 2022Walk These Ways: Tuning Robot Control for Generalization with Multiplicity of Behaviorcited by 138
  10. 2023Learning a Diffusion Model Policy from Rewards via Q-Score Matchingcited by 138
  11. 2017Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Modelscited by 128
  12. 2023Policy Representation via Diffusion Probability Model for Reinforcement Learningcited by 112
  13. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous controlcited by 108
  14. 2020Mirror Descent Policy Optimizationcited by 106
  15. 2023Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learningcited by 100
  16. 2019CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicitycited by 95
  17. 2024Simple Hierarchical Planning with Diffusioncited by 73
  18. 2018How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experimentscited by 60
  19. 2023Empirical Design in Reinforcement Learningcited by 55
  20. 2013The Information Geometry of Mirror Descentcited by 54
  21. 2025Much Ado About Noising: Dispelling the Myths of Generative Robotic Controlcited by 36
  22. 2025Scaling Offline RL via Efficient and Expressive Shortcut Modelscited by 27
  23. 2025Langevin Soft Actor-Critic: Efficient Exploration through Uncertainty-Driven Critic Learningcited by 17
  24. 2025Divergence-Augmented Policy Optimizationcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Mean Flow Policy Optimization
  2. 2026Score-Based One-step MeanFlow Policy Optimization
  3. 2026From Suboptimal to Optimal: One-Step Latent Flow Policies for Offline Reinforcement Learning
  4. 2026Flow Matching Policy Optimization with Mirror Descent and Entropy Constraints
  5. 2026Truncated Rectified Flow Policy for Reinforcement Learning with One-Step Sampling
  6. 2026Flow Matching with Injected Noise for Offline-to-Online Reinforcement Learning
  7. 2025Training Diffusion Policies via Prior-Mapping Co-Evolution
  8. 2026Dual-Flow Reinforcement Learning with State-Aware Exploration