NFPO: Stabilized Policy Optimization of Normalizing Flow for Robotic Policy Learning
On the map: Deep reinforcement learning methods → RL with diffusion and flow models
Abstract (arXiv)
Deep Reinforcement Learning (DRL) has experienced significant advancements in recent years and has been widely used in many fields. In DRL-based robotic policy learning, however, current de facto policy parameterization is still multivariate Gaussian (with diagonal covariance matrix), which lacks the ability to model multi-modal distribution. In this work, we explore the adoption of a modern network architecture, i.e. Normalizing Flow (NF) as the policy parameterization for its ability of multi-modal modeling, closed form of log probability and low computation and memory overhead. However, naively training NF in online Reinforcement Learning (RL) usually leads to training instability. We provide a detailed analysis for this phenomenon and successfully address it via simple but effective technique. With extensive experiments in multiple simulation environments, we show our method, NFPO could obtain robust and strong performance in widely used robotic learning tasks and successfully transfer into real-world robots.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL with diffusion and flow models, one of 15 directions in Deep reinforcement learning methods. Builds on 45 of its 52 references that are in the map. First posted on arXiv 12 Mar 2026.
Builds on
45 of 52 references.
Closest ancestors · 8
Same direction, newest first
- 2026Flow Policy Gradients for Robot Control2 Feb
- 2025Flow Matching Policy Gradients28 Jul
- 2025Normalizing Flows are Capable Models for Continuous Control29 May
- 2025GenPO: Generative Diffusion Models Meet On-Policy Reinforcement Learning24 May
- 2025Maximum Entropy Reinforcement Learning with Diffusion Policy17 Feb
- 2024Diffusion Policy Policy Optimization1 Sep
- 2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization25 May
- 2024Maximum Entropy Reinforcement Learning via Energy-Based Normalizing Flow22 May
Other references · 37
- 2020Denoising Diffusion Probabilistic Modelscited by 21,076
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
- 2022Generative Adversarial Networkscited by 5,627
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
- 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
- 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
- 2015Trust Region Policy Optimizationcited by 2,580
- 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
- 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
- 2016Density estimation using Real NVPcited by 2,042
25 more
- 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
- 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
- 2024Octo: An Open-Source Generalist Robot Policycited by 1,484
- 2014NICE: Non-linear Independent Components Estimationcited by 1,058
- 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 998
- 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 604
- 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
- 2021RMA: Rapid Motor Adaptation for Legged Robotscited by 508
- 2022Learning robust perceptive locomotion for quadrupedal robots in the wildcited by 382
- 2023Orbit: A Unified Simulation Framework for Interactive Robot Learning Environmentscited by 295
- 2020Why Normalizing Flows Fail to Detect Out-of-Distribution Datacited by 221
- 2024OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learningcited by 189
- 2023Real-World Humanoid Locomotion with Reinforcement Learningcited by 187
- 2022Rapid Locomotion via Reinforcement Learningcited by 168
- 2025BeyondMimic: From Motion Tracking to Versatile Humanoid Control via Guided Diffusioncited by 165
- 2023Learning a Diffusion Model Policy from Rewards via Q-Score Matchingcited by 138
- 2022Deep Whole-Body Control: Learning a Unified Policy for Manipulation and Locomotioncited by 121
- 2025TWIST: Teleoperated Whole-Body Imitation Systemcited by 101
- 2024Normalizing Flows are Capable Generative Modelscited by 91
- 2025RSL-RL: A Learning Library for Robotics Researchcited by 49
- 2025FastTD3: Simple, Fast, and Capable Reinforcement Learning for Humanoid Controlcited by 44
- 2025FALCON: Learning Force-Adaptive Humanoid Loco-Manipulationcited by 40
- 2023Robust and Versatile Bipedal Jumping Control through Reinforcement Learningcited by 36
- 2025LangWBC: Language-directed Humanoid Whole-Body Control via End-to-end Learningcited by 22
- 2024Stable Training of Normalizing Flows for High-dimensional Variational Inferencecited by 6
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026PolicyFlow: Policy Optimization with Continuous Normalizing Flow in Reinforcement Learning
- 2025An Investigation of Batch Normalization in Off-Policy Actor-Critic Algorithms
- 2025Opinion: Towards Unified Expressive Policy Optimization for Robust Robot Learning
- 2025ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learning
- 2024Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes
- 2024Design of Restricted Normalizing Flow towards Arbitrary Stochastic Policy with Computational Efficiency
- 2026Physics-Informed Policy Optimization via Analytic Dynamics Regularization
- 2026Mean Flow Policy Optimization