# Improving Exploration in Soft-Actor-Critic with Normalizing Flows Policies

arXiv: https://arxiv.org/abs/1906.02771 · HTML: https://mlsanity.com/paper/1906.02771/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Jun 2019.

## Built on by (most cited first, 30 of 38)

- [Guided Flows for Generative Modeling and Decision Making](https://mlsanity.com/paper/2311.13443/) (2023)
- [Learning Robust Autonomous Navigation and Locomotion for Wheeled-Legged Robots](https://mlsanity.com/paper/2405.01792/) (2024)
- [Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergences](https://mlsanity.com/paper/2107.08285/) (2021)
- [Universal approximation property of invertible neural networks](https://mlsanity.com/paper/2204.07415/) (2022)
- [Maximum Entropy Reinforcement Learning via Energy-Based Normalizing Flow](https://mlsanity.com/paper/2405.13629/) (2024)
- [Action Noise in Off-Policy Deep Reinforcement Learning: Impact on Exploration and Performance](https://mlsanity.com/paper/2206.03787/) (2022)
- [Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directions](https://mlsanity.com/paper/2402.13777/) (2024)
- [Flow-based Recurrent Belief State Learning for POMDPs](https://mlsanity.com/paper/2205.11051/) (2022)
- [FlowPG: Action-constrained Policy Gradient with Normalizing Flows](https://mlsanity.com/paper/2402.05149/) (2024)
- [Generative Models in Decision Making: A Survey](https://mlsanity.com/paper/2502.17100/) (2025)
- [Learning a subspace of policies for online adaptation in Reinforcement Learning](https://mlsanity.com/paper/2110.05169/) (2021)
- [Revisiting Discrete Soft Actor-Critic](https://mlsanity.com/paper/2209.10081/) (2022)
- [Latent State Marginalization as a Low-cost Approach for Improving Exploration](https://mlsanity.com/paper/2210.00999/) (2022)
- [Generative Modelling of Stochastic Actions with Arbitrary Constraints in Reinforcement Learning](https://mlsanity.com/paper/2311.15341/) (2023)
- [Implicitly Regularized RL with Implicit Q-Values](https://mlsanity.com/paper/2108.07041/) (2021)
- [Monotone, Bi-Lipschitz, and Polyak-Lojasiewicz Networks](https://mlsanity.com/paper/2402.01344/) (2024)
- [Improved Soft Actor-Critic: Mixing Prioritized Off-Policy Samples with On-Policy Experience](https://mlsanity.com/paper/2109.11767/) (2021)
- [TO-FLOW: Efficient Continuous Normalizing Flows with Temporal Optimization adjoint with Moving Speed](https://mlsanity.com/paper/2203.10335/) (2022)
- [Trust Region Policy Optimization with Optimal Transport Discrepancies: Duality and Algorithm for Continuous Actions](https://mlsanity.com/paper/2210.11137/) (2022)
- [Robust normalizing flows using Bernstein-type polynomials](https://mlsanity.com/paper/2102.03509/) (2021)
- [Reward Bonuses with Gain Scheduling Inspired by Iterative Deepening Search](https://mlsanity.com/paper/2212.10765/) (2022)
- [CEIP: Combining Explicit and Implicit Priors for Reinforcement Learning with Demonstrations](https://mlsanity.com/paper/2210.09496/) (2022)
- [Soft Actor-Critic Algorithm with Truly-satisfied Inequality Constraint](https://mlsanity.com/paper/2303.04356/) (2023)
- [How to Choose a Reinforcement-Learning Algorithm](https://mlsanity.com/paper/2407.20917/) (2024)
- [Fail-Safe Adversarial Generative Imitation Learning](https://mlsanity.com/paper/2203.01696/) (2022)
- [Leveraging Constraint Violation Signals For Action-Constrained Reinforcement Learning](https://mlsanity.com/paper/2502.10431/) (2025)
- [NFTR: From Provable Mode-Averaging to Geodesic Subgoal Selection in Offline Goal-Conditioned RL](https://mlsanity.com/paper/2607.07855/) (2026)
- [On Learning the Tail Quantiles of Driving Behavior Distributions via Quantile Regression and Flows](https://mlsanity.com/paper/2305.13106/) (2023)
- [Towards Interpretable Reinforcement Learning with Constrained Normalizing Flow Policies](https://mlsanity.com/paper/2405.01198/) (2024)
- [Design of Restricted Normalizing Flow towards Arbitrary Stochastic Policy with Computational Efficiency](https://mlsanity.com/paper/2412.12894/) (2024)

## Abstract (arXiv)

Deep Reinforcement Learning (DRL) algorithms for continuous action spaces are known to be brittle toward hyperparameters as well as cutbeingsample inefficient. Soft Actor Critic (SAC) proposes an off-policy deep actor critic algorithm within the maximum entropy RL framework which offers greater stability and empirical gains. The choice of policy distribution, a factored Gaussian, is motivated by cutchosen dueits easy re-parametrization rather than its modeling power. We introduce Normalizing Flow policies within the SAC framework that learn more expressive classes of policies than simple factored Gaussians. cutWe also present a series of stabilization tricks that enable effective training of these policies in the RL setting.We show empirically on continuous grid world tasks that our approach increases stability and is better suited to difficult exploration in sparse reward settings.