mlsanityData to 14 Oct 2026

floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RL

arXiv 2509.06863 ↗First posted 8 Sep 2025cs.LG cs.AIcited by 22

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

A hallmark of modern large-scale machine learning techniques is the use of training objectives that provide dense supervision to intermediate computations, such as teacher forcing the next token in language models or denoising step-by-step in diffusion models. This enables models to learn complex functions in a generalizable manner. Motivated by this observation, we investigate the benefits of iterative computation for temporal difference (TD) methods in reinforcement learning (RL). Typically they represent value functions in a monolithic fashion, without iterative compute. We introduce floq (flow-matching Q-functions), an approach that parameterizes the Q-function using a velocity field and trains it using techniques from flow-matching, typically used in generative modeling. This velocity field underneath the flow is trained using a TD-learning objective, which bootstraps from values produced by a target velocity field, computed by running multiple steps of numerical integration. Crucially, floq allows for more fine-grained control and scaling of the Q-function capacity than monolithic architectures, by appropriately setting the number of integration steps. Across a suite of challenging offline RL benchmarks and online fine-tuning tasks, floq improves performance by nearly 1.8x. floq scales capacity far better than standard TD-learning architectures, highlighting the potential of iterative computation for value learning.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 77 of its 85 references that are in the map. First posted on arXiv 8 Sep 2025.

Builds on

77 of 85 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learning18 Jun
  2. 2025Flow Q-Learning4 Feb
  3. 2024The Ingredients for Robotic Diffusion Transformers14 Oct
  4. 2024Diffusion Policy Policy Optimization1 Sep
  5. 2024Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradient2 Jun
  6. 2023Policy Representation via Diffusion Probability Model for Reinforcement Learning22 May
  7. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  8. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 69

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  6. 2022Flow Matching for Generative Modelingcited by 5,702
  7. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  8. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  9. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  10. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  11. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  12. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
38 more
  1. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  2. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  4. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  5. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  6. 2022Building Normalizing Flows with Stochastic Interpolantscited by 995
  7. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  8. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  9. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  10. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  11. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
  12. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  13. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  14. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  15. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  16. 2022Temporal Difference Learning for Model Predictive Controlcited by 397
  17. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRscited by 377
  18. 2018Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learningcited by 276
  19. 2022Cold Diffusion: Inverting Arbitrary Image Transforms Without Noisecited by 276
  20. 2020Critic Regularized Regressioncited by 275
  21. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 253
  22. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 238
  23. 2021Mastering Atari Games with Limited Datacited by 237
  24. 2022Multi-Game Decision Transformerscited by 214
  25. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 144
  26. 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
  27. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 131
  28. 2020Model-Based Offline Planningcited by 129
  29. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 126
  30. 2023Extreme Q-Learning: MaxEnt RL without Entropycited by 116
  31. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 116
  32. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous controlcited by 108
  33. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RLcited by 103
  34. 2019CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicitycited by 95
  35. 2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 89
  36. 2024Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbonecited by 79
  37. 2022Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trialscited by 78
  38. 2024Mixtures of Experts Unlock Parameter Scaling for Deep RLcited by 78

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 21.

22 papers.

Sort:

By area:

  1. 2026TQL: Scaling Q-Functions with Transformers by Preventing Attention Collapsecited by 9
  2. 2026Q-Flow: Stable and Expressive Reinforcement Learning with Flow-Based Policycited by 6
  3. 2026DeFlow: Decoupling Manifold Modeling and Value Maximization for Offline Policy Extractioncited by 5
  4. 2026Reinforcement Learning via Value Gradient Flowcited by 5
  5. 2026Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matchingcited by 5
  6. 2026ReFORM: Reflected Flows for On-support Offline RL via Noise Manipulationcited by 4
  7. 2026GoldenStart: Q-Guided Priors and Entropy Control for Distilling Flow Policiescited by 3
  8. 2026Causal Flow Q-Learning for Robust Offline Reinforcement Learningcited by 2
  9. 2026What Does Flow Matching Bring To TD Learning?cited by 2
  10. 2026Refining Compositional Diffusion for Reliable Long-Horizon Planningcited by 2
  11. 2026Path-Coupled Bellman Flows for Distributional Reinforcement Learningcited by 2
  12. 2026Reversal Q-Learningcited by 2
10 more
  1. 2026Posterior Augmented Flow Matchingcited by 1
  2. 2026Quantile-Coupled Flow Matching for Distributional Reinforcement Learningcited by 1
  3. 2025RAMAC: Multimodal Risk-Aware Offline Reinforcement Learning and the Role of Behavior Regularizationcited by 0
  4. 2026Distributional Reinforcement Learning with Diffusion Bridge Criticscited by 0
  5. 2026DFPO: Scaling Value Modeling via Distributional Flow towards Robust and Generalizable LLM Post-Trainingcited by 0
  6. 2026Horizon Adaptive Offline Policy Learning via Value Stitchingcited by 0
  7. 2026Retimed Bellman Flows: Escaping the Impossible Triangle of Velocity Bootstrappingcited by 0
  8. 2026Flowing Faster to Coordinate: One-Step Online Multi-Agent Flow Policiescited by 0
  9. 2026LASER: Latent Space Adjoint Matching for Support-Constrained Entropy-Regularized Offline RLcited by 0
  10. 2026Posterior-Averaged Bellman Flow Matching for Distributional Criticscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026A Survey of Flow Matching in Reinforcement Learning
  2. 2025One-Step Flow Q-Learning: Addressing the Diffusion Policy Bottleneck in Offline Reinforcement Learning
  3. 2026Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learning
  4. 2026Reinforcement Learning via Fixed-point Flow Matching
  5. 2026Q-Learning with Scalar Adjoint Matching
  6. 2026Reverse Flow Matching: A Unified Framework for Online Reinforcement Learning with Diffusion and Flow Policies
  7. 2026Flow Matching with Injected Noise for Offline-to-Online Reinforcement Learning
  8. 2026Q-VGM: Q-Guided Value-Gradient Matching for Offline-to-Online RL of Flow-Matching VLA Policies