# Controllable Flow Matching for Online Reinforcement Learning

arXiv: https://arxiv.org/abs/2511.06816 · HTML: https://mlsanity.com/paper/2511.06816/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models.
- Builds on 22 of its 34 references that are in the map.
- First posted on arXiv 10 Nov 2025.

## Placed in

- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)
- Deep reinforcement learning methods / [RL with diffusion and flow models](https://mlsanity.com/deeprl/rl-with-diffusion-and-flow-models/)

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [Planning with Diffusion for Flexible Behavior Synthesis](https://mlsanity.com/paper/2205.09991/) (2022)
- [Building Normalizing Flows with Stochastic Interpolants](https://mlsanity.com/paper/2209.15571/) (2022)
- [Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models](https://mlsanity.com/paper/1805.12114/) (2018)
- [When to Trust Your Model: Model-Based Policy Optimization](https://mlsanity.com/paper/1906.08253/) (2019)
- [Model-Ensemble Trust-Region Policy Optimization](https://mlsanity.com/paper/1802.10592/) (2018)
- [Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion](https://mlsanity.com/paper/1807.01675/) (2018)
- [AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Planners](https://mlsanity.com/paper/2302.01877/) (2023)
- [Consistency Flow Matching: Defining Straight Flows with Velocity Consistency](https://mlsanity.com/paper/2407.02398/) (2024)
- [Diffusion Actor-Critic with Entropy Regulator](https://mlsanity.com/paper/2405.15177/) (2024)
- [Combating the Compounding-Error Problem with a Multi-step Model](https://mlsanity.com/paper/1905.13320/) (2019)
- [On the Guidance of Flow Matching](https://mlsanity.com/paper/2502.02150/) (2025)
- [Learning Transformer-based World Models with Contrastive Predictive Coding](https://mlsanity.com/paper/2503.04416/) (2025)
- [Model-based Offline Reinforcement Learning with Lower Expectile Q-Learning](https://mlsanity.com/paper/2407.00699/) (2024)
- [Model-Based Control with Sparse Neural Dynamics](https://mlsanity.com/paper/2312.12791/) (2023)
- [Informed POMDP: Leveraging Additional Information in Model-Based RL](https://mlsanity.com/paper/2306.11488/) (2023)
- [Learning to Reach Goals via Diffusion](https://mlsanity.com/paper/2310.02505/) (2023)
- [Model-based Reinforcement Learning with Multi-step Plan Value Estimation](https://mlsanity.com/paper/2209.05530/) (2022)
- [Control analysis and synthesis for general control-affine systems](https://mlsanity.com/paper/2503.05606/) (2025)

## Abstract (arXiv)

Model-based reinforcement learning (MBRL) typically relies on modeling environment dynamics for data efficiency. However, due to the accumulation of model errors over long-horizon rollouts, such methods often face challenges in maintaining modeling stability. To address this, we propose CtrlFlow, a trajectory-level synthetic method using conditional flow matching (CFM), which directly modeling the distribution of trajectories from initial states to high-return terminal states without explicitly modeling the environment transition function. Our method ensures optimal trajectory sampling by minimizing the control energy governed by the non-linear Controllability Gramian Matrix, while the generated diverse trajectory data significantly enhances the robustness and cross-task generalization of policy learning. In online settings, CtrlFlow demonstrates the better performance on common MuJoCo benchmark tasks than dynamics models and achieves superior sample efficiency compared to standard MBRL methods.