# One-Step Generative Policies with Q-Learning: A Reformulation of MeanFlow

arXiv: https://arxiv.org/abs/2511.13035 · HTML: https://mlsanity.com/paper/2511.13035/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for offline RL, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 26 of its 41 references that are in the map.
- First posted on arXiv 17 Nov 2025.

## Placed in

- Offline RL and imitation learning from data / [Diffusion and flow policies for offline RL](https://mlsanity.com/offlinerl/diffusion-and-flow-policies-for-offline-rl/)

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [Octo: An Open-Source Generalist Robot Policy](https://mlsanity.com/paper/2405.12213/) (2024)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [Offline Reinforcement Learning with Implicit Q-Learning](https://mlsanity.com/paper/2110.06169/) (2021)
- [AWAC: Accelerating Online Reinforcement Learning with Offline Datasets](https://mlsanity.com/paper/2006.09359/) (2020)
- [Mean Flows for One-step Generative Modeling](https://mlsanity.com/paper/2505.13447/) (2025)
- [Improved Techniques for Training Consistency Models](https://mlsanity.com/paper/2310.14189/) (2023)
- [One Step Diffusion via Shortcut Models](https://mlsanity.com/paper/2410.12557/) (2024)
- [IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies](https://mlsanity.com/paper/2304.10573/) (2023)
- [Consistency Policy: Accelerated Visuomotor Policies via Consistency Distillation](https://mlsanity.com/paper/2405.07503/) (2024)
- [CORL: Research-oriented Deep Offline Reinforcement Learning Library](https://mlsanity.com/paper/2210.07105/) (2022)
- [FlowPolicy: Enabling Fast and Robust 3D Flow-based Policy via Consistency Flow Matching for Robot Manipulation](https://mlsanity.com/paper/2412.04987/) (2024)
- [Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learning](https://mlsanity.com/paper/2309.16984/) (2023)
- [Diffusion Actor-Critic with Entropy Regulator](https://mlsanity.com/paper/2405.15177/) (2024)
- [Is Value Learning Really the Main Bottleneck in Offline RL?](https://mlsanity.com/paper/2406.09329/) (2024)
- [Score Regularized Policy Optimization through Diffusion Behavior](https://mlsanity.com/paper/2310.07297/) (2023)
- [Generative Pre-training for Speech with Flow Matching](https://mlsanity.com/paper/2310.16338/) (2023)
- [MP1: MeanFlow Tames Policy Learning in 1-step for Robotic Manipulation](https://mlsanity.com/paper/2507.10543/) (2025)
- [Provable Guarantees for Generative Behavior Cloning: Bridging Low-Level Stability and High-Level Behavior](https://mlsanity.com/paper/2307.14619/) (2023)
- [Scaling Offline RL via Efficient and Expressive Shortcut Models](https://mlsanity.com/paper/2505.22866/) (2025)
- [Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directions](https://mlsanity.com/paper/2402.13777/) (2024)
- [Flow-Based Single-Step Completion for Efficient and Expressive Policy Learning](https://mlsanity.com/paper/2506.21427/) (2025)
- [Revisiting Generative Policies: A Simpler Reinforcement Learning Algorithmic Perspective](https://mlsanity.com/paper/2412.01245/) (2024)

## Built on by (most cited first, 18 of 18)

- [DeFlow: Decoupling Manifold Modeling and Value Maximization for Offline Policy Extraction](https://mlsanity.com/paper/2601.10471/) (2026)
- [Drifting Field Policy: A One-Step Generative Policy via Wasserstein Gradient Flow](https://mlsanity.com/paper/2605.07727/) (2026)
- [QPILOTS: Efficient Test-Time Q-Steering for Flow Policies](https://mlsanity.com/paper/2606.14801/) (2026)
- [Latent Policy Steering through One-Step Flow Policies](https://mlsanity.com/paper/2603.05296/) (2026)
- [Ada3Drift: Adaptive Training-Time Drifting for One-Step 3D Visuomotor Robotic Manipulation](https://mlsanity.com/paper/2603.11984/) (2026)
- [Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learning](https://mlsanity.com/paper/2605.01663/) (2026)
- [Drift Flow Matching](https://mlsanity.com/paper/2605.17244/) (2026)
- [Fast and Highly Expressive Policy Learning for Offline Reinforcement Learning via Bootstrapped Flow Q-Learning](https://mlsanity.com/paper/2606.10613/) (2026)
- [Path-Space Mirror Descent for On-Policy Reinforcement Learning under the Generalized Schr"odinger Bridge](https://mlsanity.com/paper/2603.21621/) (2026)
- [Preserve Support, Not Correspondence: Dynamic Routing for Offline Reinforcement Learning](https://mlsanity.com/paper/2604.22229/) (2026)
- [Discrete MeanFlow: One-Step Generation via Conditional Transition Kernels](https://mlsanity.com/paper/2605.12805/) (2026)
- [Stochastic MeanFlow Policies: One-Step Generative Control with Entropic Mirror Descent](https://mlsanity.com/paper/2605.21282/) (2026)
- [ReFPO: Reflow Regularization for Flow Matching Policy Gradients](https://mlsanity.com/paper/2606.21086/) (2026)
- [CoDrift: Compositional Drifting for Offline Reinforcement Learning](https://mlsanity.com/paper/2608.23939/) (2026)
- [Kinematic MeanFlow: One-Step Action Generation Policy for Robotic Foundation Models](https://mlsanity.com/paper/2610.00864/) (2026)
- [Flowing Faster to Coordinate: One-Step Online Multi-Agent Flow Policies](https://mlsanity.com/paper/2610.01882/) (2026)
- [Compositional Average Velocity Modeling for Efficient Diffusion Offline Reinforcement Learning](https://mlsanity.com/paper/2610.14941/) (2026)
- [From Suboptimal to Optimal: One-Step Latent Flow Policies for Offline Reinforcement Learning](https://mlsanity.com/paper/2610.16984/) (2026)

## Abstract (arXiv)

We introduce a one-step generative policy for offline reinforcement learning that maps noise directly to actions via a residual reformulation of MeanFlow, making it compatible with Q-learning. While one-step Gaussian policies enable fast inference, they struggle to capture complex, multimodal action distributions. Existing flow-based methods improve expressivity but typically rely on distillation and two-stage training when trained with Q-learning. To overcome these limitations, we propose to reformulate MeanFlow to enable direct noise-to-action generation by integrating the velocity field and noise-to-action transformation into a single policy network-eliminating the need for separate velocity estimation. We explore several reformulation variants and identify an effective residual formulation that supports expressive and stable policy learning. Our method offers three key advantages: 1) efficient one-step noise-to-action generation, 2) expressive modelling of multimodal action distributions, and 3) efficient and stable policy learning via Q-learning in a single-stage training setup. Extensive experiments on 73 tasks across the OGBench and D4RL benchmarks demonstrate that our method achieves strong performance in both offline and offline-to-online reinforcement learning settings. Code is available at https://github.com/HiccupRL/MeanFlowQL.