mlsanityData to 7 Oct 2026

Maximum Entropy Reinforcement Learning via Energy-Based Normalizing Flow

arXiv 2405.13629 ↗First posted 22 May 2024cs.LGcited by 25

Not on the map. Why?

Abstract (arXiv)

Existing Maximum-Entropy (MaxEnt) Reinforcement Learning (RL) methods for continuous action spaces are typically formulated based on actor-critic frameworks and optimized through alternating steps of policy evaluation and policy improvement. In the policy evaluation steps, the critic is updated to capture the soft Q-function. In the policy improvement steps, the actor is adjusted in accordance with the updated soft Q-function. In this paper, we introduce a new MaxEnt RL framework modeled using Energy-Based Normalizing Flows (EBFlow). This framework integrates the policy evaluation steps and the policy improvement steps, resulting in a single objective training process. Our method enables the calculation of the soft value function used in the policy evaluation target without Monte Carlo approximation. Moreover, this design supports the modeling of multi-modal action distributions while facilitating efficient action sampling. To evaluate the performance of our method, we conducted experiments on the MuJoCo benchmark suite and a number of high-dimensional robotic tasks simulated by Omniverse Isaac Gym. The evaluation results demonstrate that our method achieves superior performance compared to widely-adopted representative baselines.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 46 of its 82 references that are in the map. First posted on arXiv 22 May 2024.

Builds on

46 of 82 references.

References · 46

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2016Layer Normalizationcited by 5,579
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2015Continuous control with deep reinforcement learningcited by 3,279
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  7. 2016Density estimation using Real NVPcited by 2,029
  8. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,539
  9. 2019Normalizing Flows for Probabilistic Modeling and Inferencecited by 1,315
  10. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  11. 2014NICE: Non-linear Independent Components Estimationcited by 1,052
  12. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
34 more
  1. 2017Masked Autoregressive Flow for Density Estimationcited by 680
  2. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  3. 2016Improving Variational Inference with Inverse Autoregressive Flowcited by 636
  4. 2019Neural Spline Flowscited by 590
  5. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 547
  6. 2016Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithmcited by 498
  7. 2019Your Classifier is Secretly an Energy Based Model and You Should Treat it Like Onecited by 402
  8. 2018Maximum a Posteriori Policy Optimisationcited by 318
  9. 2015MADE: Masked Autoencoder for Distribution Estimationcited by 306
  10. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
  11. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 169
  12. 2018Neural Importance Samplingcited by 154
  13. 2015Taming the Noise in Reinforcement Learning via Soft Updatescited by 125
  14. 2018Latent Space Policies for Hierarchical Reinforcement Learningcited by 66
  15. 2023normflows: A PyTorch Package for Normalizing Flowscited by 56
  16. 2019Leveraging exploration in off-policy algorithms via normalizing flowscited by 45
  17. 2019Improving Exploration in Soft-Actor-Critic with Normalizing Flows Policiescited by 37
  18. 2022skrl: Modular and Flexible Library for Reinforcement Learningcited by 35
  19. 2019MaCow: Masked Convolutional Generative Flowcited by 32
  20. 2019Emerging Convolutions for Generative Normalizing Flowscited by 31
  21. 2021A Statistical Analysis of Polyak-Ruppert Averaged Q-learningcited by 29
  22. 2016Combining policy gradient and Q-learningcited by 23
  23. 2016Learning to Draw Samples: With Application to Amortized MLE for Generative Adversarial Learningcited by 21
  24. 2019Tsallis Reinforcement Learning: A Unified Framework for Maximum Entropy Reinforcement Learningcited by 18
  25. 2019Soft Policy Gradient Method for Maximum Entropy Deep Reinforcement Learningcited by 18
  26. 2024S2AC: Energy-Based Reinforcement Learning with Stein Soft Actor Criticcited by 18
  27. 2020Relative gradient optimization of the Jacobian term in unsupervised deep learningcited by 16
  28. 2022Latent State Marginalization as a Low-cost Approach for Improving Explorationcited by 13
  29. 2020Self Normalizing Flowscited by 12
  30. 2022ButterflyFlow: Building Invertible Layers with Butterfly Matricescited by 12
  31. 2020Woodbury Transformations for Deep Generative Flowscited by 10
  32. 2021Robot Skill Adaptation via Soft Actor-Critic Gaussian Mixture Modelscited by 9
  33. 2022Optimistic Curiosity Exploration and Conservative Exploitation with Linear Reward Shapingcited by 6
  34. 2023Training Energy-Based Normalizing Flow with Score-Matching Objectivescited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 11; 2026: 14.

25 papers.

Sort:

By area:

  1. 2025FlowRL: Matching Reward Distributions for LLM Reasoningcited by 38
  2. 2025Maximum Entropy Reinforcement Learning with Diffusion Policycited by 29
  3. 2025GenPO: Generative Diffusion Models Meet On-Policy Reinforcement Learningcited by 16
  4. 2025Normalizing Flows are Capable Models for Continuous Controlcited by 13
  5. 2026Maximizing Local Entropy Where It Matters: Prefix-Aware Localized LLM Unlearningcited by 10
  6. 2025When Maximum Entropy Misleads Policy Optimizationcited by 7
  7. 2025Efficient Reinforcement Learning with Semantic and Token Entropy for LLM Reasoningcited by 6
  8. 2026PolicyFlow: Policy Optimization with Continuous Normalizing Flow in Reinforcement Learningcited by 5
  9. 2025Entropy Regularizing Activation: Boosting Continuous Control, Large Language Models, and Image Classification with Activation as Entropy Constraintscited by 4
  10. 2026Tracking Drift: Variation-Aware Entropy Scheduling for Non-Stationary Reinforcement Learningcited by 4
  11. 2026Expected Return Causes Outcome-Level Mode Collapse in Reinforcement Learning and How to Fix It with Inverse Probability Scalingcited by 4
  12. 2025Token-Level Policy Optimization: Linking Group-Level Rewards to Token-Level Aggregation via Markov Likelihoodcited by 3
13 more
  1. 2026Boosting Maximum Entropy Reinforcement Learning via One-Step Flow Matchingcited by 2
  2. 2026SERNF: Sample-Efficient Real-World Dexterous Policy Fine-Tuning via Action-Chunked Critics and Normalizing Flowscited by 2
  3. 2025CDFlow: Building Invertible Layers with Circulant and Diagonal Matricescited by 1
  4. 2025A Theoretical Lens for RL-Tuned Language Models via Energy-Based Modelscited by 1
  5. 2026Diffusion Policy through Conditional Proximal Policy Optimizationcited by 1
  6. 2026Mean Flow Policy Optimizationcited by 1
  7. 2026Generative Actor-Critic with Soft Bridge Policiescited by 1
  8. 2026GenPO++: Generative Policy Optimization with Jacobian-free Likelihood Ratioscited by 1
  9. 2026NFTR: From Provable Mode-Averaging to Geodesic Subgoal Selection in Offline Goal-Conditioned RLcited by 1
  10. 2025Frictional Q-Learningcited by 0
  11. 2026Flow Perturbation++: Multi-Step Unbiased Jacobian Estimation for High-Dimensional Boltzmann Samplingcited by 0
  12. 2026Harnessing Implicit Cooperation: A Multi-Agent Reinforcement Learning Approach Towards Decentralized Local Energy Marketscited by 0
  13. 2026Safety-Critical Contextual Control via Online Riemannian Optimization with World Modelscited by 0