mlsanityData to 7 Oct 2026

Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learning

arXiv 2605.01663 ↗First posted 3 May 2026cs.LG cs.ROcited by 0

On the map: Offline RL and imitation learning from data → Diffusion and flow policies for offline RL

Abstract (arXiv)

We propose Flow-Anchored Noise-conditioned Q-Learning (FAN), a highly efficient and high-performing offline reinforcement learning (RL) algorithm. Recent work has shown that expressive flow policies and distributional critics improve offline RL performance, but at a high computational cost. Specifically, flow policies require iterative sampling to produce a single action, and distributional critics require computation over multiple samples (e.g., quantiles) to estimate value. To address these inefficiencies while maintaining high performance, we introduce FAN. Our method employs a behavior regularization technique that uses a single flow policy iteration and requires a single Gaussian noise sample for distributional critics. Our theoretical analysis of convergence and performance bounds demonstrates that these simplifications not only improve efficiency but also lead to superior task performance. Experiments on robotic manipulation and locomotion tasks demonstrate that FAN achieves state-of-the-art performance while significantly reducing both training and inference runtimes. We release our code at https://github.com/brianlsy98/FAN.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Diffusion and flow policies for offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 54 of its 79 references that are in the map. First posted on arXiv 3 May 2026.

Builds on

54 of 79 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025Guided Flow Policy: Learning from High-Value Actions in Offline Reinforcement Learning3 Dec
  2. 2025One-Step Generative Policies with Q-Learning: A Reformulation of MeanFlow17 Nov
  3. 2025Energy-Weighted Flow Matching for Offline Reinforcement Learning6 Mar
  4. 2025Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning7 Feb
  5. 2025Flow Q-Learning4 Feb
  6. 2024OGBench: Benchmarking Offline Goal-Conditioned RL26 Oct
  7. 2024Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learning29 Jul
  8. 2024Aligning Diffusion Behaviors with Q-functions for Efficient Continuous Control12 Jul
  9. 2024Diffusion Policies creating a Trust Region for Offline Reinforcement Learning30 May
  10. 2023Score Regularized Policy Optimization through Diffusion Behavior11 Oct
  11. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  12. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modeling29 Sep

Other references · 42

  1. 2022Flow Matching for Generative Modelingcited by 5,426
  2. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  3. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  5. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  7. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  8. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  9. 2022Building Normalizing Flows with Stochastic Interpolantscited by 964
  10. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  11. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  12. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
30 more
  1. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  2. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  3. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  4. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  5. 2018A Finite Time Analysis of Temporal Difference Learning With Linear Function Approximationcited by 225
  6. 2019Finite-Time Error Bounds For Linear Stochastic Approximation and TD Learningcited by 165
  7. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 136
  8. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learningcited by 125
  9. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 121
  10. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Librarycited by 117
  11. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 114
  12. 2023Extreme Q-Learning: MaxEnt RL without Entropycited by 113
  13. 2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimizationcited by 99
  14. 2021Conservative Offline Distributional Reinforcement Learningcited by 78
  15. 2021Risk-Averse Offline Reinforcement Learningcited by 65
  16. 2019Statistics and Samples in Distributional Reinforcement Learningcited by 62
  17. 2025Horizon Reduction Makes RL Scalablecited by 55
  18. 2023Anti-Exploration by Random Network Distillationcited by 49
  19. 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learningcited by 48
  20. 2023Reasoning with Latent Diffusion in Offline Reinforcement Learningcited by 33
  21. 2023The Benefits of Being Distributional: Small-Loss Bounds for Reinforcement Learningcited by 27
  22. 2025Scaling Offline RL via Efficient and Expressive Shortcut Modelscited by 27
  23. 2024More Benefits of Being Distributional: Second-Order Bounds for Reinforcement Learningcited by 23
  24. 2023Trust Region-Based Safe Distributional Reinforcement Learning for Multiple Constraintscited by 20
  25. 2024AlignIQL: Policy Alignment in Implicit Q-Learning through Constrained Optimizationcited by 19
  26. 2025Value Flowscited by 12
  27. 2025Scalable Offline Model-Based RL with Action Chunkscited by 12
  28. 2025Multi-agent Coordination via Flow Matchingcited by 9
  29. 2025Expressive Value Learning for Scalable Offline Reinforcement Learningcited by 6
  30. 2025Temporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learningcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026ReFORM: Reflected Flows for On-support Offline RL via Noise Manipulation
  2. 2026Flow Matching with Injected Noise for Offline-to-Online Reinforcement Learning
  3. 2026Fast and Highly Expressive Policy Learning for Offline Reinforcement Learning via Bootstrapped Flow Q-Learning
  4. 2026Flow Actor-Critic for Offline Reinforcement Learning
  5. 2026Reversal Q-Learning
  6. 2025Offline Reinforcement Learning with Penalized Action Noise Injection
  7. 2026QF3: Fast Flow RL with Filtered Q-Gradients
  8. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functions