# PrT-Flow: Probability-Tractable Flow Matching for Reinforcement Learning

arXiv: https://arxiv.org/abs/2610.17667 · HTML: https://mlsanity.com/paper/2610.17667/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models.
- Builds on 27 of its 29 references that are in the map.
- First posted on arXiv 13 Oct 2026.

## Placed in

- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)
- Deep reinforcement learning methods / [RL with diffusion and flow models](https://mlsanity.com/deeprl/rl-with-diffusion-and-flow-models/)

## Builds on (most cited first, 27 of 27)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [What Matters in Learning from Offline Human Demonstrations for Robot Manipulation](https://mlsanity.com/paper/2108.03298/) (2021)
- [Neural Spline Flows](https://mlsanity.com/paper/1906.04032/) (2019)
- [Flow Matching Guide and Code](https://mlsanity.com/paper/2412.06264/) (2024)
- [Diffusion Policy Policy Optimization](https://mlsanity.com/paper/2409.00588/) (2024)
- [OGBench: Benchmarking Offline Goal-Conditioned RL](https://mlsanity.com/paper/2410.20092/) (2024)
- [Flow Q-Learning](https://mlsanity.com/paper/2502.02538/) (2025)
- [Flow Matching Policy Gradients](https://mlsanity.com/paper/2507.21053/) (2025)
- [Normalizing Flows are Capable Generative Models](https://mlsanity.com/paper/2412.06329/) (2024)
- [Learning Robotic Manipulation Policies from Point Clouds with Conditional Flow Matching](https://mlsanity.com/paper/2409.07343/) (2024)
- [Leveraging exploration in off-policy algorithms via normalizing flows](https://mlsanity.com/paper/1905.06893/) (2019)
- [Improving Exploration in Soft-Actor-Critic with Normalizing Flows Policies](https://mlsanity.com/paper/1906.02771/) (2019)
- [Sliced Iterative Normalizing Flows](https://mlsanity.com/paper/2007.00674/) (2020)
- [SAC Flow: Sample-Efficient Reinforcement Learning of Flow-Based Policies via Velocity-Reparameterized Sequential Modeling](https://mlsanity.com/paper/2509.25756/) (2025)
- [FALCON: Few-step Accurate Likelihoods for Continuous Flows](https://mlsanity.com/paper/2512.09914/) (2025)
- [Efficient Regression-Based Training of Normalizing Flows for Boltzmann Generators](https://mlsanity.com/paper/2506.01158/) (2025)
- [SERNF: Sample-Efficient Real-World Dexterous Policy Fine-Tuning via Action-Chunked Critics and Normalizing Flows](https://mlsanity.com/paper/2602.09580/) (2026)
- [GenPO++: Generative Policy Optimization with Jacobian-free Likelihood Ratios](https://mlsanity.com/paper/2606.06967/) (2026)
- [Reinforcement Learning for Flow-Matching Policies with Density Transport](https://mlsanity.com/paper/2606.08602/) (2026)
- [Max-Entropy Reinforcement Learning with Flow Matching and A Case Study on LQR](https://mlsanity.com/paper/2512.23870/) (2025)
- [NFPO: Stabilized Policy Optimization of Normalizing Flow for Robotic Policy Learning](https://mlsanity.com/paper/2603.11470/) (2026)

## Abstract (arXiv)

Diffusion models and Flow Matching can represent flexible distributions over robot actions, but many reinforcement learning algorithms also require the probability of each action. We introduce PrT-Flow (Probability-Tractable Flow Matching), which computes probability density while retaining multimodal, multi-step action generation. Its Jacobian-Tractable Transformer (JTT) makes each sampling step invertible and enables efficient computation of its Jacobian determinant. We train these finite steps with Conditional Euler-step Training (CET), which supervises their displacement over one step during sampling rather than their instantaneous velocity. We prove that, under certain assumptions, CET has the same minimizers as fitting the Euler-solver updates of Flow Matching. PrT-Flow achieves 96% success on Robomimic Lift after 300 epochs of imitation learning, matching the diffusion policy reference, and 96% success on OGBench Cube-Triple after 600k online steps. These results demonstrate that invertible networks with tractable probability densities can learn effective robot policies and support SAC-style actor-critic training. Our code is available at https://github.com/Sunlight-zero/PrT-Flow .