# Boosting Maximum Entropy Reinforcement Learning via One-Step Flow Matching

arXiv: https://arxiv.org/abs/2602.01606 · HTML: https://mlsanity.com/paper/2602.01606/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models.
- Builds on 26 of its 31 references that are in the map.
- First posted on arXiv 2 Feb 2026.

## Placed in

- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)
- Deep reinforcement learning methods / [RL with diffusion and flow models](https://mlsanity.com/deeprl/rl-with-diffusion-and-flow-models/)

## Builds on (most cited first, 26 of 26)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [DeepMind Control Suite](https://mlsanity.com/paper/1801.00690/) (2018)
- [Reinforcement Learning with Deep Energy-Based Policies](https://mlsanity.com/paper/1702.08165/) (2017)
- [FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models](https://mlsanity.com/paper/1810.01367/) (2018)
- [Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels](https://mlsanity.com/paper/2004.13649/) (2020)
- [Mean Flows for One-step Generative Modeling](https://mlsanity.com/paper/2505.13447/) (2025)
- [Learning a Diffusion Model Policy from Rewards via Q-Score Matching](https://mlsanity.com/paper/2312.11752/) (2023)
- [Policy Representation via Diffusion Probability Model for Reinforcement Learning](https://mlsanity.com/paper/2305.13122/) (2023)
- [Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization](https://mlsanity.com/paper/2405.16173/) (2024)
- [Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learning](https://mlsanity.com/paper/2309.16984/) (2023)
- [ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learning](https://mlsanity.com/paper/2505.22094/) (2025)
- [Diffusion Actor-Critic with Entropy Regulator](https://mlsanity.com/paper/2405.15177/) (2024)
- [Error Bounds for Flow Matching Methods](https://mlsanity.com/paper/2305.16860/) (2023)
- [AdaFlow: Imitation Learning with Variance-Adaptive Flow-Based Policies](https://mlsanity.com/paper/2402.04292/) (2024)
- [Online Reward-Weighted Fine-Tuning of Flow Matching with Wasserstein Regularization](https://mlsanity.com/paper/2502.06061/) (2025)
- [Flow-Based Policy for Online Reinforcement Learning](https://mlsanity.com/paper/2506.12811/) (2025)
- [Maximum Entropy Reinforcement Learning with Diffusion Policy](https://mlsanity.com/paper/2502.11612/) (2025)
- [Maximum Entropy Reinforcement Learning via Energy-Based Normalizing Flow](https://mlsanity.com/paper/2405.13629/) (2024)

## Built on by (most cited first, 2 of 2)

- [Flow Matching Policy Optimization with Mirror Descent and Entropy Constraints](https://mlsanity.com/paper/2603.17685/) (2026)
- [Constrained Flow Policy Updates: A Generalized Schr"odinger Bridge View](https://mlsanity.com/paper/2609.32952/) (2026)

## Abstract (arXiv)

Diffusion policies are expressive yet incur high inference latency. Flow Matching (FM) enables one-step generation, but integrating it into Maximum Entropy Reinforcement Learning (MaxEnt RL) is challenging: the optimal policy is an intractable energy-based distribution, and the efficient log-likelihood estimation required to balance exploration and exploitation suffers from severe discretization bias. We propose Flow-based Log-likelihood-Aware Maximum Entropy RL (FLAME), a principled framework that addresses these challenges. First, we derive a Q-Reweighted FM objective that bypasses partition function estimation via importance reweighting. Second, we design a decoupled entropy estimator that rigorously corrects bias, which enables efficient exploration and brings the policy closer to the optimal MaxEnt policy. Third, we integrate the MeanFlow formulation to achieve expressive and efficient one-step control. Empirical results on MuJoCo show that FLAME outperforms Gaussian baselines and matches multi-step diffusion policies with significantly lower inference cost. Code is available at https://github.com/lzqw/FLAME.