mlsanityData to 7 Oct 2026

Max-Entropy Reinforcement Learning with Flow Matching and A Case Study on LQR

arXiv 2512.23870 ↗First posted 29 Dec 2025cs.LGcited by 0

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

Soft actor-critic (SAC) is a popular algorithm for max-entropy reinforcement learning. In practice, the energy-based policies in SAC are often approximated using simple policy classes for efficiency, sacrificing the expressiveness and robustness. In this paper, we propose a variant of the SAC algorithm that parameterizes the policy with flow-based models, leveraging their rich expressiveness. In the algorithm, we evaluate the flow-based policy utilizing the instantaneous change-of-variable technique and update the policy with an online variant of flow matching developed in this paper. This online variant, termed importance sampling flow matching (ISFM), enables policy update with only samples from a user-specified sampling distribution rather than the unknown target distribution. We develop a theoretical analysis of ISFM, characterizing how different choices of sampling distributions affect the learning efficiency. Finally, we conduct a case study of our algorithm on the max-entropy linear quadratic regulator problems, demonstrating that the proposed algorithm learns the optimal action distribution.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 16 of its 26 references that are in the map. First posted on arXiv 29 Dec 2025.

Builds on

16 of 26 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Efficient Online Reinforcement Learning for Diffusion Policy1 Feb
  2. 2024Diffusion Actor-Critic with Entropy Regulator24 May
  3. 2023Policy Representation via Diffusion Probability Model for Reinforcement Learning22 May
  4. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr

Other references · 12

  1. 2022Flow Matching for Generative Modelingcited by 5,426
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2019Normalizing Flows for Probabilistic Modeling and Inferencecited by 1,315
  4. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  5. 2019A Theory of Regularized Markov Decision Processescited by 279
  6. 2021How to Train Your Energy-Based Modelscited by 271
  7. 2017A unified view of entropy-regularized Markov decision processescited by 174
  8. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 169
  9. 2019Leveraging exploration in off-policy algorithms via normalizing flowscited by 45
  10. 2025An Introduction to Flow Matching and Diffusion Modelscited by 33
  11. 2024Convergence of Continuous Normalizing Flows for Learning Probability Distributionscited by 24
  12. 2023Fast Policy Learning for Linear Quadratic Control with Entropy Regularizationcited by 11

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Boosting Maximum Entropy Reinforcement Learning via One-Step Flow Matching
  2. 2025Maximum Entropy Reinforcement Learning with Diffusion Policy
  3. 2026Truncated Rectified Flow Policy for Reinforcement Learning with One-Step Sampling
  4. 2020Revisiting Maximum Entropy Inverse Reinforcement Learning: New Perspectives and Algorithms
  5. 2026Generative Actor-Critic with Soft Bridge Policies
  6. 2026Flow Matching Policy Optimization with Mirror Descent and Entropy Constraints
  7. 2026Mean Flow Policy Optimization
  8. 2025DIME:Diffusion-Based Maximum Entropy Reinforcement Learning