mlsanityData to 7 Oct 2026

Flow Actor-Critic for Offline Reinforcement Learning

arXiv 2602.18015 ↗First posted 20 Feb 2026cs.LG cs.AIcited by 6

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

The dataset distributions in offline reinforcement learning (RL) often exhibit complex and multi-modal distributions, necessitating expressive policies to capture such distributions beyond widely-used Gaussian policies. To handle such complex and multi-modal datasets, in this paper, we propose Flow Actor-Critic, a new actor-critic method for offline RL, based on recent flow policies. The proposed method not only uses the flow model for actor as in previous flow policies but also exploits the expressive flow model for conservative critic acquisition to prevent Q-value explosion in out-of-data regions. To this end, we propose a new form of critic regularizer based on the flow behavior proxy model obtained as a byproduct of flow-based actor design. Leveraging the flow model in this joint way, we achieve new state-of-the-art performance for test datasets of offline RL including the D4RL and recent OGBench benchmarks.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 67 of its 85 references that are in the map. First posted on arXiv 20 Feb 2026.

Builds on

67 of 85 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024AlignIQL: Policy Alignment in Implicit Q-Learning through Constrained Optimization28 May
  2. 2024Exclusively Penalized Q-learning for Offline Reinforcement Learning23 May
  3. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learning16 May
  4. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning9 Mar
  5. 2023Anti-Exploration by Random Network Distillation31 Jan
  6. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
  7. 2022Supported Policy Optimization for Offline Reinforcement Learning13 Feb
  8. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  9. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation21 Jun
  10. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  11. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  12. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
2 more
  1. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  2. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 53

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  6. 2016Layer Normalizationcited by 5,579
  7. 2022Flow Matching for Generative Modelingcited by 5,426
  8. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  9. 2018Neural Ordinary Differential Equationscited by 3,733
  10. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  11. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  12. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
38 more
  1. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  2. 2016Density estimation using Real NVPcited by 2,029
  3. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  5. 2014NICE: Non-linear Independent Components Estimationcited by 1,052
  6. 2018Exploration by Random Network Distillationcited by 966
  7. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  8. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  9. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  10. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  11. 2018FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Modelscited by 584
  12. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  13. 2021Tackling the Generative Learning Trilemma with Denoising Diffusion GANscited by 515
  14. 2017Robust Adversarial Reinforcement Learningcited by 439
  15. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  16. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 307
  17. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 216
  18. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 177
  19. 2019A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptationcited by 166
  20. 2025Flow Q-Learningcited by 162
  21. 2023Efficient Diffusion Policies for Offline Reinforcement Learningcited by 151
  22. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learningcited by 145
  23. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RLcited by 103
  24. 2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimizationcited by 99
  25. 2023Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learningcited by 93
  26. 2017Learning Invariant Feature Spaces to Transfer Skills with Reinforcement Learningcited by 84
  27. 2025Energy-Weighted Flow Matching for Offline Reinforcement Learningcited by 70
  28. 2022Policy Gradient Method For Robust Reinforcement Learningcited by 67
  29. 2023Score Regularized Policy Optimization through Diffusion Behaviorcited by 67
  30. 2022Matching Normalizing Flows and Probability Paths on Manifoldscited by 57
  31. 2024Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learningcited by 29
  32. 2023DiffCPS: Diffusion Model based Constrained Policy Search for Offline Reinforcement Learningcited by 27
  33. 2022Robust Imitation Learning against Variations in Environment Dynamicscited by 26
  34. 2023Decision ConvFormer: Local Filtering in MetaFormer is Sufficient for Decision Makingcited by 26
  35. 2024Reinformer: Max-Return Sequence Modeling for Offline RLcited by 25
  36. 2018Estimating the Spectral Density of Large Implicit Matricescited by 16
  37. 2024Decision Mamba: A Multi-Grained State Space Model with Self-Evolution Regularization for Offline RLcited by 15
  38. 2024The Max-Min Formulation of Multi-Objective Reinforcement Learning: From Theory to a Model-Free Algorithmcited by 13

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Adaptive Action Chunking via Multi-Chunk Q Value Estimationcited by 4
  2. 2026Drift Q-Learningcited by 2
  3. 2026SPAR: Support-Preserving Action Rectificationcited by 1
  4. 2026Preserve Support, Not Correspondence: Dynamic Routing for Offline Reinforcement Learningcited by 0
  5. 2026Bridging Domain Gaps with Target-Aligned Generation for Offline Reinforcement Learningcited by 0
  6. 2026Convex-Hull-Neighborhood Smooth Dual Generalization: Controlling Local Correction Propagation in Offline RLcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Simple Actors and Deep Critics for Scalable Reinforcement Learning
  2. 2026Decoupling Policy Extraction for Offline Reinforcement Learning
  3. 2025Unleashing Flow Policies with Distributional Critics
  4. 2025Guided Flow Policy: Learning from High-Value Actions in Offline Reinforcement Learning
  5. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning
  6. 2023Importance Weighted Actor-Critic for Optimal Conservative Offline Reinforcement Learning
  7. 2021Offline RL Without Off-Policy Evaluation
  8. 2024Offline Actor-Critic Reinforcement Learning Scales to Large Models