mlsanityData to 14 Oct 2026

Normalizing Flows are Capable Models for Continuous Control

arXiv 2505.23527 ↗First posted 29 May 2025cs.LGcited by 14

On the map: Deep reinforcement learning methods → RL with diffusion and flow models

Abstract (arXiv)

Modern reinforcement learning (RL) algorithms have found success by using powerful probabilistic models, such as transformers, energy-based models, and diffusion/flow-based models. To this end, RL researchers often choose to pay the price of accommodating these models into their algorithms -- diffusion models are expressive, but are computationally intensive due to their reliance on solving differential equations, while autoregressive transformer models are scalable but typically require learning discrete representations. Normalizing flows (NFs), by contrast, seem to provide an appealing alternative, as they enable likelihoods and sampling without solving differential equations or autoregressive architectures. However, their potential in RL has received limited attention, partly due to the prevailing belief that normalizing flows lack sufficient expressivity. We show that this is not the case. Building on recent work in NFs, we propose a single NF architecture which integrates seamlessly into RL algorithms, serving as a policy, Q-function, and occupancy measure. Our approach leads to much simpler algorithms, and achieves higher performance in imitation learning, offline, goal conditioned RL and unsupervised RL.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in RL with diffusion and flow models, one of 15 directions in Deep reinforcement learning methods. Builds on 80 of its 93 references that are in the map. First posted on arXiv 29 May 2025.

Builds on

80 of 93 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Flow Q-Learning4 Feb
  2. 2024Diffusion Policy Policy Optimization1 Sep
  3. 2024Maximum Entropy Reinforcement Learning via Energy-Based Normalizing Flow22 May

Other references · 77

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  6. 2022Classifier-Free Diffusion Guidancecited by 5,847
  7. 2022Flow Matching for Generative Modelingcited by 5,702
  8. 2016Layer Normalizationcited by 5,669
  9. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  10. 2017Neural Discrete Representation Learningcited by 4,712
  11. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  12. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
38 more
  1. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,407
  2. 2015Variational Inference with Normalizing Flowscited by 2,170
  3. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  4. 2016Density estimation using Real NVPcited by 2,042
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  6. 2023Consistency Modelscited by 1,822
  7. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  8. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  9. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,545
  10. 2016Variational Inference: A Review for Statisticianscited by 1,472
  11. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  12. 2019Normalizing Flows for Probabilistic Modeling and Inferencecited by 1,323
  13. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  14. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  15. 2023Mastering Diverse Domains through World Modelscited by 1,253
  16. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  17. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  18. 2014NICE: Non-linear Independent Components Estimationcited by 1,058
  19. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  20. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  21. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  22. 2017Masked Autoregressive Flow for Density Estimationcited by 684
  23. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  24. 2016Improving Variational Inference with Inverse Autoregressive Flowcited by 640
  25. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 516
  26. 2019Model-Based Reinforcement Learning for Ataricited by 502
  27. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  28. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 459
  29. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 395
  30. 2021Flow Network based Generative Models for Non-Iterative Diverse Candidate Generationcited by 392
  31. 2022Transformers are Sample-Efficient World Modelscited by 328
  32. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 325
  33. 2022Behavior Transformers: Cloning k modes with one stonecited by 304
  34. 2021GFlowNet Foundationscited by 290
  35. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 274
  36. 2020Decoupling Representation Learning from Reinforcement Learningcited by 269
  37. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 259
  38. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 253

50 most cited of 77 All 77 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 12.

14 papers.

Sort:

By area:

  1. 2026Q-Flow: Stable and Expressive Reinforcement Learning with Flow-Based Policycited by 6
  2. 2025NinA: Normalizing Flows in Action. Training VLA Models with Normalizing Flowscited by 5
  3. 2026SERNF: Sample-Efficient Real-World Dexterous Policy Fine-Tuning via Action-Chunked Critics and Normalizing Flowscited by 3
  4. 2026GoldenStart: Q-Guided Priors and Entropy Control for Distilling Flow Policiescited by 3
  5. 2026Causal Flow Q-Learning for Robust Offline Reinforcement Learningcited by 2
  6. 2026NFTR: From Provable Mode-Averaging to Geodesic Subgoal Selection in Offline Goal-Conditioned RLcited by 2
  7. 2025Beyond Distributions: Geometric Action Control for Continuous Reinforcement Learningcited by 1
  8. 2026Data-Efficient Hierarchical Goal-Conditioned Reinforcement Learning via Normalizing Flowscited by 1
  9. 2026NFPO: Stabilized Policy Optimization of Normalizing Flow for Robotic Policy Learningcited by 1
  10. 2026Maximum Entropy Behavior Exploration for Sim2Real Zero-Shot Reinforcement Learningcited by 1
  11. 2026Warp RL: Reshaping Base Policy Distributions for Dynamics Adaptationcited by 1
  12. 2026DFPO: Scaling Value Modeling via Distributional Flow towards Robust and Generalizable LLM Post-Trainingcited by 0
2 more
  1. 2026Escaping Offline Pessimism: Vector-Field Reward Shaping for Safe Frontier Explorationcited by 0
  2. 2026QHyer: Q-conditioned Hybrid Attention-mamba Transformer for Offline Goal-conditioned RLcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026A Survey of Flow Matching in Reinforcement Learning
  2. 2026PolicyFlow: Policy Optimization with Continuous Normalizing Flow in Reinforcement Learning
  3. 2024Design of Restricted Normalizing Flow towards Arbitrary Stochastic Policy with Computational Efficiency
  4. 2022Let Offline RL Flow: Training Conservative Agents in the Latent Space of Normalizing Flows
  5. 2026Flow Matching for Offline Reinforcement Learning with Discrete Actions
  6. 2026Reinforcement Learning for Flow-Matching Policies with Density Transport
  7. 2026FlowRL: A Taxonomy and Modular Framework for Reinforcement Learning with Diffusion Policies
  8. 2023A Coupled Flow Approach to Imitation Learning