mlsanityData to 14 Oct 2026

Flow Q-Learning

arXiv 2502.02538 ↗First posted 4 Feb 2025cs.LG cs.AIcited by 184

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 2 more
Abstract (arXiv)

We present flow Q-learning (FQL), a simple and performant offline reinforcement learning (RL) method that leverages an expressive flow-matching policy to model arbitrarily complex action distributions in data. Training a flow policy with RL is a tricky problem, due to the iterative nature of the action generation process. We address this challenge by training an expressive one-step policy with RL, rather than directly guiding an iterative flow policy to maximize values. This way, we can completely avoid unstable recursive backpropagation, eliminate costly iterative action generation at test time, yet still mostly maintain expressivity. We experimentally show that FQL leads to strong performance across 73 challenging state- and pixel-based OGBench and D4RL tasks in offline RL and offline-to-online RL. Project page: https://seohong.me/projects/fql/

Key facts. Cited by 184 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 79 of its 95 references that are in the map. First posted on arXiv 4 Feb 2025.

Builds on

79 of 95 references.

Closest ancestors · 23

Same direction, newest first

  1. 2024Diffusion Policy Policy Optimization1 Sep
  2. 2024Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learning29 Jul
  3. 2024Aligning Diffusion Behaviors with Q-functions for Efficient Continuous Control12 Jul
  4. 2024Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradient2 Jun
  5. 2024Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning31 May
  6. 2024Diffusion Policies creating a Trust Region for Offline Reinforcement Learning30 May
  7. 2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization25 May
  8. 2024Policy-Guided Diffusion9 Apr
  9. 2024Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learning6 Feb
  10. 2024Diffusion World Model: Future Modeling Beyond Step-by-Step Rollout for Offline Reinforcement Learning5 Feb
  11. 2024Simple Hierarchical Planning with Diffusion5 Jan
  12. 2023Learning a Diffusion Model Policy from Rewards via Q-Score Matching18 Dec
11 more
  1. 2023Score Regularized Policy Optimization through Diffusion Behavior11 Oct
  2. 2023Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learning29 Sep
  3. 2023Reasoning with Latent Diffusion in Offline Reinforcement Learning12 Sep
  4. 2023Diffusion Policies for Out-of-Distribution Generalization in Offline Reinforcement Learning10 Jul
  5. 2023Efficient Diffusion Policies for Offline Reinforcement Learning31 May
  6. 2023Policy Representation via Diffusion Probability Model for Reinforcement Learning22 May
  7. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learning25 Apr
  8. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  9. 2023AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Planners3 Feb
  10. 2022Is Conditional Generative Modeling all you need for Decision-Making?28 Nov
  11. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 56

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  3. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  5. 2022Flow Matching for Generative Modelingcited by 5,702
  6. 2016Layer Normalizationcited by 5,669
  7. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  8. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  9. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  10. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  11. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  12. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,407
38 more
  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  4. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  5. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  6. 2022Building Normalizing Flows with Stochastic Interpolantscited by 995
  7. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  8. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  9. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 862
  10. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  11. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 725
  12. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  13. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  14. 2023InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generationcited by 395
  15. 2024One Step Diffusion via Shortcut Modelscited by 391
  16. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  17. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 302
  18. 2024Flow Matching Guide and Codecited by 301
  19. 2024Diffusion for World Modeling: Visual Details Matter in Ataricited by 300
  20. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 238
  21. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 237
  22. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 214
  23. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 182
  24. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 161
  25. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 126
  26. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
  27. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Librarycited by 119
  28. 2023Extreme Q-Learning: MaxEnt RL without Entropycited by 116
  29. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 116
  30. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous controlcited by 108
  31. 2023Synthetic Experience Replaycited by 101
  32. 2023Guided Flows for Generative Modeling and Decision Makingcited by 101
  33. 2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 89
  34. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 86
  35. 2024Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbonecited by 79
  36. 2024SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learningcited by 76
  37. 2023Anti-Exploration by Random Network Distillationcited by 50
  38. 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learningcited by 48

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 52; 2026: 132.

184 papers.

Sort:

By area:

  1. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 169
  2. 2025Reinforcement Learning with Action Chunkingcited by 107
  3. 2025A Vision-Language-Action-Critic Model for Robotic Real-World Reinforcement Learningcited by 71
  4. 2025RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learningcited by 59
  5. 2025Diffusion Guidance Is a Controllable Policy Improvement Operatorcited by 58
  6. 2025Horizon Reduction Makes RL Scalablecited by 57
  7. 2026RL Token: Bootstrapping Online RL with Vision-Language-Action Modelscited by 49
  8. 2025FlowRL: Matching Reward Distributions for LLM Reasoningcited by 40
  9. 2025Flow-Based Policy for Online Reinforcement Learningcited by 37
  10. 2026Q-learning with Adjoint Matchingcited by 37
  11. 2025EXPO: Stable Reinforcement Learning with Expressive Policiescited by 34
  12. 2025SAC Flow: Sample-Efficient Reinforcement Learning of Flow-Based Policies via Velocity-Reparameterized Sequential Modelingcited by 31
38 more
  1. 2025Scaling Offline RL via Efficient and Expressive Shortcut Modelscited by 27
  2. 2025Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learnerscited by 23
  3. 2025floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RLcited by 22
  4. 2025TD-JEPA: Latent-predictive Representations for Zero-Shot Reinforcement Learningcited by 22
  5. 2025One-Step Flow Policy Mirror Descentcited by 21
  6. 2025GenPO: Generative Diffusion Models Meet On-Policy Reinforcement Learningcited by 20
  7. 2025Dual Goal Representationscited by 16
  8. 2026Flow Policy Gradients for Robot Controlcited by 16
  9. 2025Value Flowscited by 15
  10. 2026Learning While Deploying: Fleet-Scale Reinforcement Learning for Generalist Robot Policiescited by 15
  11. 2025Prior-Guided Diffusion Planning for Offline Reinforcement Learningcited by 14
  12. 2025Normalizing Flows are Capable Models for Continuous Controlcited by 14
  13. 2025Flow-Based Single-Step Completion for Efficient and Expressive Policy Learningcited by 14
  14. 2025Scalable Offline Model-Based RL with Action Chunkscited by 13
  15. 2025Graph-Assisted Stitching for Offline Hierarchical Reinforcement Learningcited by 12
  16. 2025Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agentscited by 12
  17. 2025Transitive RL: Value Learning via Divide and Conquercited by 12
  18. 2026Reverse Flow Matching: A Unified Framework for Online Reinforcement Learning with Diffusion and Flow Policiescited by 12
  19. 2025One-Step Flow Q-Learning: Addressing the Diffusion Policy Bottleneck in Offline Reinforcement Learningcited by 11
  20. 2025DEAS: DEtached value learning with Action Sequence for Scalable Offline RLcited by 10
  21. 2025Multi-agent Coordination via Flow Matchingcited by 10
  22. 2026From Prior to Pro: Efficient Skill Mastery via Distribution Contractive RL Finetuningcited by 10
  23. 2025Intention-Conditioned Flow Occupancy Modelscited by 9
  24. 2025Guided Flow Policy: Learning from High-Value Actions in Offline Reinforcement Learningcited by 9
  25. 2026TQL: Scaling Q-Functions with Transformers by Preventing Attention Collapsecited by 9
  26. 2026Mean Flow Policy with Instantaneous Velocity Constraint for One-step Action Generationcited by 9
  27. 2025Unleashing Flow Policies with Distributional Criticscited by 8
  28. 2026Compositional Planning with Jumpy World Modelscited by 7
  29. 2025Expressive Value Learning for Scalable Offline Reinforcement Learningcited by 6
  30. 2025Dichotomous Diffusion Policy Optimizationcited by 6
  31. 2026FLAC: Maximum Entropy RL via Kinetic Energy Regularized Bridge Matchingcited by 6
  32. 2026Flow Actor-Critic for Offline Reinforcement Learningcited by 6
  33. 2026Q-Flow: Stable and Expressive Reinforcement Learning with Flow-Based Policycited by 6
  34. 2026Test-Time Gradient Guidance of Flow Policies in Reinforcement Learningcited by 6
  35. 2025Human-in-the-loop Online Rejection Sampling for Robotic Manipulationcited by 5
  36. 2026How Does the Lagrangian Guide Safe Reinforcement Learning through Diffusion Models?cited by 5
  37. 2026Finite Difference Flow Optimization for RL Post-Training of Text-to-Image Modelscited by 5
  38. 2026Reinforcement Learning via Value Gradient Flowcited by 5

50 most cited of 184 All 184 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Flow Matching for Offline Reinforcement Learning with Discrete Actions
  2. 2025One-Step Generative Policies with Q-Learning: A Reformulation of MeanFlow
  3. 2026A Survey of Flow Matching in Reinforcement Learning
  4. 2026QF3: Fast Flow RL with Filtered Q-Gradients
  5. 2025FlowQ: Energy-Guided Flow Policies for Offline Reinforcement Learning
  6. 2025Behavior-Adaptive Q-Learning: A Unifying Framework for Offline-to-Online RL
  7. 2025Pretraining a Shared Q-Network for Data-Efficient Offline Reinforcement Learning
  8. 2026Refinement-based Flow Policy Optimization