# Design of Restricted Normalizing Flow towards Arbitrary Stochastic Policy with Computational Efficiency

arXiv: https://arxiv.org/abs/2412.12894 · HTML: https://mlsanity.com/paper/2412.12894/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 70 references that are in the map.
- First posted on arXiv 17 Dec 2024.

## Builds on (most cited first, 30 of 40)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1702.03118/) (2017)
- [When to Trust Your Model: Model-Based Policy Optimization](https://mlsanity.com/paper/1906.08253/) (2019)
- [Reinforcement Learning with Deep Energy-Based Policies](https://mlsanity.com/paper/1702.08165/) (2017)
- [Deep Exploration via Bootstrapped DQN](https://mlsanity.com/paper/1602.04621/) (2016)
- [End-to-end Driving via Conditional Imitation Learning](https://mlsanity.com/paper/1710.02410/) (2017)
- [Robust Adversarial Reinforcement Learning](https://mlsanity.com/paper/1703.02702/) (2017)
- [StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flows](https://mlsanity.com/paper/2008.02401/) (2020)
- [A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms](https://mlsanity.com/paper/1907.03146/) (2019)
- [Coupling-based Invertible Neural Networks Are Universal Diffeomorphism Approximators](https://mlsanity.com/paper/2006.11469/) (2020)
- [Understanding and Mitigating Exploding Inverses in Invertible Neural Networks](https://mlsanity.com/paper/2006.09347/) (2020)
- [Partially Observable Markov Decision Processes (POMDPs) and Robotics](https://mlsanity.com/paper/2107.07599/) (2021)
- [Leveraging exploration in off-policy algorithms via normalizing flows](https://mlsanity.com/paper/1905.06893/) (2019)
- [Physical Symmetries Embedded in Neural Networks](https://mlsanity.com/paper/1904.08991/) (2019)
- [Invertible Generative Modeling using Linear Rational Splines](https://mlsanity.com/paper/2001.05168/) (2020)
- [Variational Inference MPC for Bayesian Model-based Reinforcement Learning](https://mlsanity.com/paper/1907.04202/) (2019)
- [Improving Exploration in Soft-Actor-Critic with Normalizing Flows Policies](https://mlsanity.com/paper/1906.02771/) (2019)
- [AdvFlow: Inconspicuous Black-box Adversarial Attacks using Normalizing Flows](https://mlsanity.com/paper/2007.07435/) (2020)
- [The Expressive Power of a Class of Normalizing Flow Models](https://mlsanity.com/paper/2006.00392/) (2020)
- [PlaNet of the Bayesians: Reconsidering and Improving Deep Planning Network by Incorporating Bayesian Inference](https://mlsanity.com/paper/2003.00370/) (2020)
- [Sigsoftmax: Reanalysis of the Softmax Bottleneck](https://mlsanity.com/paper/1805.10829/) (2018)
- [Squareplus: A Softplus-Like Algebraic Rectifier](https://mlsanity.com/paper/2112.11687/) (2021)

## Built on by (most cited first, 1 of 1)

- [Pseudo-Quantized Actor-Critic Algorithm for Robustness to Noisy Temporal Difference Error](https://mlsanity.com/paper/2604.01613/) (2026)

## Abstract (arXiv)

This paper proposes a new design method for a stochastic control policy using a normalizing flow (NF). In reinforcement learning (RL), the policy is usually modeled as a distribution model with trainable parameters. When this parameterization has less expressiveness, it would fail to acquiring the optimal policy. A mixture model has capability of a universal approximation, but it with too much redundancy increases the computational cost, which can become a bottleneck when considering the use of real-time robot control. As another approach, NF, which is with additional parameters for invertible transformation from a simple stochastic model as a base, is expected to exert high expressiveness and lower computational cost. However, NF cannot compute its mean analytically due to complexity of the invertible transformation, and it lacks reliability because it retains stochastic behaviors after deployment for robot controller. This paper therefore designs a restricted NF (RNF) that achieves an analytic mean by appropriately restricting the invertible transformation. In addition, the expressiveness impaired by this restriction is regained using bimodal student-t distribution as its base, so-called Bit-RNF. In RL benchmarks, Bit-RNF policy outperformed the previous models. Finally, a real robot experiment demonstrated the applicability of Bit-RNF policy to real world. The attached video is uploaded on youtube: https://youtu.be/R_GJVZDW9bk