mlsanityData to 14 Oct 2026

Drift, Then Spread: Learning One-Step Generative Policies for Online Reinforcement Learning

arXiv 2610.15662 ↗First posted 11 Oct 2026cs.LGcited by 0

On the map: Deep reinforcement learning methods → RL with diffusion and flow models also fits: RL theory and sample complexity

Abstract (arXiv)

Generative policies offer expressive action distributions for online reinforcement learning (RL), but many diffusion- and flow-based policies require costly iterative sampling. We propose DriftSpread, a one-step generative policy based on Drifting for online maximum-entropy (MaxEnt) RL. To incorporate entropy without a tractable policy density, we build on an energy-based view of Drifting and derive critic-guided Drift and entropy-induced Spread from the negative MaxEnt objective. Our method learns these updates sequentially in the same actor, using action regression for value improvement and distribution matching to capture the effect of entropy. The actor updates require neither direct samples from the critic-defined Boltzmann target nor policy-density or score estimation. Under exact fitting with a fixed critic, we prove convergence to the Boltzmann target in the small-step limit followed by the long-time limit. On MuJoCo, DriftSpread achieves the highest aggregate normalized return among all evaluated methods, while outperforming the diffusion- and flow-based baselines in both training and inference efficiency.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL with diffusion and flow models, one of 15 directions in Deep reinforcement learning methods. Builds on 24 of its 30 references that are in the map. First posted on arXiv 11 Oct 2026.

Builds on

24 of 30 references.

Closest ancestors · 13

Same direction, newest first

  1. 2026Drift Q-Learning29 May
  2. 2026Score-Based One-step MeanFlow Policy Optimization22 May
  3. 2026Mean Flow Policy Optimization16 Apr
  4. 2026Reverse Flow Matching: A Unified Framework for Online Reinforcement Learning with Diffusion and Flow Policies13 Jan
  5. 2025SAC Flow: Sample-Efficient Reinforcement Learning of Flow-Based Policies via Velocity-Reparameterized Sequential Modeling30 Sep
  6. 2025One-Step Flow Policy Mirror Descent31 Jul
  7. 2025Flow-Based Policy for Online Reinforcement Learning15 Jun
  8. 2025Maximum Entropy Reinforcement Learning with Diffusion Policy17 Feb
  9. 2025DIME:Diffusion-Based Maximum Entropy Reinforcement Learning4 Feb
  10. 2025Efficient Online Reinforcement Learning for Diffusion Policy1 Feb
  11. 2024Sampling from Energy-based Policies using Diffusion2 Oct
  12. 2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization25 May
1 more
  1. 2024Diffusion Actor-Critic with Entropy Regulator24 May

Other references · 11

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  3. 2025Mean Flows for One-step Generative Modelingcited by 561
  4. 2015Generative Moment Matching Networkscited by 257
  5. 2026Generative Modeling via Driftingcited by 116
  6. 2019A Characteristic Function Approach to Deep Implicit Generative Modelingcited by 26
  7. 2026Sinkhorn-Drifting Generative Modelscited by 24
  8. 2026One-Step Generative Modeling via Wasserstein Gradient Flowscited by 24
  9. 2026On the Wasserstein Gradient Flow Interpretation of Drifting Modelscited by 11
  10. 2026Wasserstein Proximal Policy Gradientcited by 4
  11. 2026Drifting Field Policy: A One-Step Generative Policy via Wasserstein Gradient Flowcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Positive-Only Drifting Policy Optimization
  2. 2025EXPO: Stable Reinforcement Learning with Expressive Policies
  3. 2026Truncated Rectified Flow Policy for Reinforcement Learning with One-Step Sampling
  4. 2025Training Diffusion Policies via Prior-Mapping Co-Evolution
  5. 2026Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matching
  6. 2026Drift-Based Policy Optimization: Native One-Step Policy Learning for Online Robot Control
  7. 2026Diffusing to Coordinate: Efficient Online Multi-Agent Diffusion Policies
  8. 2026Generative Actor-Critic with Soft Bridge Policies