mlsanityData to 14 Oct 2026

Leveraging exploration in off-policy algorithms via normalizing flows

arXiv 1905.06893 ↗First posted 16 May 2019cs.LG stat.MLcited by 46

Not on the map. Why?

Cited by 21 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The ability to discover approximately optimal policies in domains with sparse rewards is crucial to applying reinforcement learning (RL) in many real-world scenarios. Approaches such as neural density models and continuous exploration (e.g., Go-Explore) have been proposed to maintain the high exploration rate necessary to find high performing and generalizable policies. Soft actor-critic(SAC) is another method for improving exploration that aims to combine efficient learning via off-policy updates while maximizing the policy entropy. In this work, we extend SAC to a richer class of probability distributions (e.g., multimodal) through normalizing flows (NF) and show that this significantly improves performance by accelerating the discovery of good policies while using much smaller policy representations. Our approach, which we call SAC-NF, is a simple, efficient,easy-to-implement modification and improvement to SAC on continuous control baselines such as MuJoCo and PyBullet Roboschool domains. Finally, SAC-NF does this while being significantly parameter efficient, using as few as 5.5% the parameters for an equivalent SAC model.

Key facts. Cited by 46 papers in the map (data through 14 Oct 2026). First posted on arXiv 16 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 7; 2022: 9; 2023: 2; 2024: 12; 2025: 9; 2026: 7.

46 papers.

Sort:

By area:

  1. 2025Flow Q-Learningcited by 184
  2. 2024Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbonecited by 79
  3. 2021A Max-Min Entropy Framework for Reinforcement Learningcited by 28
  4. 2024Maximum Entropy Reinforcement Learning via Energy-Based Normalizing Flowcited by 26
  5. 2021Transflower: probabilistic autoregressive dance generation with multimodal attentioncited by 22
  6. 2022Action Noise in Off-Policy Deep Reinforcement Learning: Impact on Exploration and Performancecited by 22
  7. 2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
  8. 2025GenPO: Generative Diffusion Models Meet On-Policy Reinforcement Learningcited by 20
  9. 2024S2AC: Energy-Based Reinforcement Learning with Stein Soft Actor Criticcited by 18
  10. 2022Flow-based Recurrent Belief State Learning for POMDPscited by 16
  11. 2024FlowPG: Action-constrained Policy Gradient with Normalizing Flowscited by 16
  12. 2022Learning Sampling Distributions for Model Predictive Controlcited by 10
34 more
  1. 2023Generative Modelling of Stochastic Actions with Arbitrary Constraints in Reinforcement Learningcited by 10
  2. 2024Universal Functional Regression with Neural Operator Flowscited by 9
  3. 2021Measuring Progress in Deep Reinforcement Learning Sample Efficiencycited by 8
  4. 2022TO-FLOW: Efficient Continuous Normalizing Flows with Temporal Optimization adjoint with Moving Speedcited by 8
  5. 2025Unleashing Flow Policies with Distributional Criticscited by 8
  6. 2024RICE: Breaking Through the Training Bottlenecks of Reinforcement Learning with Explanationcited by 7
  7. 2025When Maximum Entropy Misleads Policy Optimizationcited by 7
  8. 2021Robust normalizing flows using Bernstein-type polynomialscited by 6
  9. 2022AUTM Flow: Atomic Unrestricted Time Machine for Monotonic Normalizing Flowscited by 5
  10. 2022Efficiently Learning Small Policies for Locomotion and Manipulationcited by 4
  11. 2022CEIP: Combining Explicit and Implicit Priors for Reinforcement Learning with Demonstrationscited by 4
  12. 2024Deep Exploration with PAC-Bayescited by 4
  13. 2025Learning Intractable Multimodal Policies with Reparameterization and Diversity Regularizationcited by 3
  14. 2026SERNF: Sample-Efficient Real-World Dexterous Policy Fine-Tuning via Action-Chunked Critics and Normalizing Flowscited by 3
  15. 2026GenPO++: Generative Policy Optimization with Jacobian-free Likelihood Ratioscited by 3
  16. 2021Improving Actor-Critic Reinforcement Learning via Hamiltonian Monte Carlo Methodcited by 2
  17. 2021Bellman: A Toolbox for Model-Based Reinforcement Learning in TensorFlowcited by 2
  18. 2022Policy Gradient With Serial Markov Chain Reasoningcited by 2
  19. 2023PACER: A Fully Push-forward-based Distributional Reinforcement Learning Algorithmcited by 2
  20. 2024Stein Variational Evolution Strategiescited by 2
  21. 2025Leveraging Constraint Violation Signals For Action-Constrained Reinforcement Learningcited by 2
  22. 2022Guided Exploration in Reinforcement Learning via Monte Carlo Critic Optimizationcited by 1
  23. 2024Towards Interpretable Reinforcement Learning with Constrained Normalizing Flow Policiescited by 1
  24. 2024Design of Restricted Normalizing Flow towards Arbitrary Stochastic Policy with Computational Efficiencycited by 1
  25. 2025Beyond Distributions: Geometric Action Control for Continuous Reinforcement Learningcited by 1
  26. 2025Max-Entropy Reinforcement Learning with Flow Matching and A Case Study on LQRcited by 1
  27. 2026Data-Efficient Hierarchical Goal-Conditioned Reinforcement Learning via Normalizing Flowscited by 1
  28. 2026Revisiting Mixture Policies in Entropy-Regularized Actor-Criticcited by 1
  29. 2021Out-of-Distribution Detection of Melanoma using Normalizing Flowscited by 0
  30. 2024Soft Actor-Critic with Beta Policy via Implicit Reparameterization Gradientscited by 0
  31. 2025Failure Modes of Maximum Entropy RLHFcited by 0
  32. 2026APC-RL: Exceeding Data-Driven Behavior Priors with Adaptive Policy Compositioncited by 0
  33. 2026Explore Beyond the Boundary Using Entropic Informationcited by 0
  34. 2026PrT-Flow: Probability-Tractable Flow Matching for Reinforcement Learningcited by 0