Design of Restricted Normalizing Flow towards Arbitrary Stochastic Policy with Computational Efficiency
Not on the map. Why?
Abstract (arXiv)
This paper proposes a new design method for a stochastic control policy using a normalizing flow (NF). In reinforcement learning (RL), the policy is usually modeled as a distribution model with trainable parameters. When this parameterization has less expressiveness, it would fail to acquiring the optimal policy. A mixture model has capability of a universal approximation, but it with too much redundancy increases the computational cost, which can become a bottleneck when considering the use of real-time robot control. As another approach, NF, which is with additional parameters for invertible transformation from a simple stochastic model as a base, is expected to exert high expressiveness and lower computational cost. However, NF cannot compute its mean analytically due to complexity of the invertible transformation, and it lacks reliability because it retains stochastic behaviors after deployment for robot controller. This paper therefore designs a restricted NF (RNF) that achieves an analytic mean by appropriately restricting the invertible transformation. In addition, the expressiveness impaired by this restriction is regained using bimodal student-t distribution as its base, so-called Bit-RNF. In RL benchmarks, Bit-RNF policy outperformed the previous models. Finally, a real robot experiment demonstrated the applicability of Bit-RNF policy to real world. The attached video is uploaded on youtube: https://youtu.be/R_GJVZDW9bk
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 40 of its 70 references that are in the map. First posted on arXiv 17 Dec 2024.
Builds on
40 of 70 references.
References · 40
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2016Layer Normalizationcited by 5,669
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
- 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
- 2016Density estimation using Real NVPcited by 2,042
- 2016OpenAI Gymcited by 1,736
- 2019Normalizing Flows for Probabilistic Modeling and Inferencecited by 1,323
- 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
- 2015Prioritized Experience Replaycited by 1,099
- 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 950
- 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
- 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
28 more
- 2016Deep Exploration via Bootstrapped DQNcited by 527
- 2017End-to-end Driving via Conditional Imitation Learningcited by 485
- 2017Robust Adversarial Reinforcement Learningcited by 451
- 2020StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flowscited by 349
- 2019A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithmscited by 142
- 2020Coupling-based Invertible Neural Networks Are Universal Diffeomorphism Approximatorscited by 85
- 2020Understanding and Mitigating Exploding Inverses in Invertible Neural Networkscited by 71
- 2021Partially Observable Markov Decision Processes (POMDPs) and Roboticscited by 49
- 2019Leveraging exploration in off-policy algorithms via normalizing flowscited by 46
- 2019Physical Symmetries Embedded in Neural Networkscited by 40
- 2020Invertible Generative Modeling using Linear Rational Splinescited by 40
- 2019Variational Inference MPC for Bayesian Model-based Reinforcement Learningcited by 39
- 2019Improving Exploration in Soft-Actor-Critic with Normalizing Flows Policiescited by 38
- 2020AdvFlow: Inconspicuous Black-box Adversarial Attacks using Normalizing Flowscited by 36
- 2020The Expressive Power of a Class of Normalizing Flow Modelscited by 35
- 2020PlaNet of the Bayesians: Reconsidering and Improving Deep Planning Network by Incorporating Bayesian Inferencecited by 23
- 2018Sigsoftmax: Reanalysis of the Softmax Bottleneckcited by 20
- 2021Squareplus: A Softplus-Like Algebraic Rectifiercited by 19
- 2020t-Soft Update of Target Network for Deep Reinforcement Learningcited by 17
- 2018TD-Regularized Actor-Critic Methodscited by 14
- 2022Variational Inference MPC using Normalizing Flows and Out-of-Distribution Projectioncited by 13
- 2020Proximal Policy Optimization with Relative Pearson Divergencecited by 9
- 2019dpVAEs: Fixing Sample Generation for Regularized VAEscited by 6
- 2022Proximal Policy Optimization with Adaptive Threshold for Symmetric Relative Density Ratiocited by 5
- 2021Maximum Entropy Reinforcement Learning with Mixture Policiescited by 4
- 2019Transflow Learning: Repurposing Flow Models Without Retrainingcited by 3
- 2021Variational Policy Search using Sparse Gaussian Process Priors for Learning Multimodal Optimal Actionscited by 3
- 2021Optimization Algorithm for Feedback and Feedforward Policies towards Robot Control Robust to Sensing Failurescited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025Normalizing Flows are Capable Models for Continuous Control
- 2026NFPO: Stabilized Policy Optimization of Normalizing Flow for Robotic Policy Learning
- 2026PolicyFlow: Policy Optimization with Continuous Normalizing Flow in Reinforcement Learning
- 2024FlowPG: Action-constrained Policy Gradient with Normalizing Flows
- 2024Maximum Entropy Reinforcement Learning via Energy-Based Normalizing Flow
- 2026Data-Efficient Hierarchical Goal-Conditioned Reinforcement Learning via Normalizing Flows
- 2026Truncated Rectified Flow Policy for Reinforcement Learning with One-Step Sampling
- 2026Flow-based Policy With Distributional Reinforcement Learning in Trajectory Optimization