# Discrete Flow Matching Policy Optimization

arXiv: https://arxiv.org/abs/2604.06491 · HTML: https://mlsanity.com/paper/2604.06491/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models.
- Builds on 24 of its 35 references that are in the map.
- First posted on arXiv 7 Apr 2026.

## Placed in

- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)
- Deep reinforcement learning methods / [RL with diffusion and flow models](https://mlsanity.com/deeprl/rl-with-diffusion-and-flow-models/)

## Builds on (most cited first, 24 of 24)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Structured Denoising Diffusion Models in Discrete State-Spaces](https://mlsanity.com/paper/2107.03006/) (2021)
- [Training Diffusion Models with Reinforcement Learning](https://mlsanity.com/paper/2305.13301/) (2023)
- [Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions](https://mlsanity.com/paper/2102.05379/) (2021)
- [A Continuous Time Framework for Discrete Denoising Models](https://mlsanity.com/paper/2205.14987/) (2022)
- [Discrete Flow Matching](https://mlsanity.com/paper/2407.15595/) (2024)
- [Generative Flows on Discrete State-Spaces: Enabling Multimodal Flows with Applications to Protein Co-Design](https://mlsanity.com/paper/2402.04997/) (2024)
- [Flow Matching Guide and Code](https://mlsanity.com/paper/2412.06264/) (2024)
- [d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning](https://mlsanity.com/paper/2504.12216/) (2025)
- [Score-based Continuous-time Discrete Diffusion Models](https://mlsanity.com/paper/2211.16750/) (2022)
- [DeFoG: Discrete Flow Matching for Graph Generation](https://mlsanity.com/paper/2410.04263/) (2024)
- [Fine-Tuning Discrete Diffusion Models via Reward Optimization with Applications to DNA and Protein Design](https://mlsanity.com/paper/2410.13643/) (2024)
- [Flow Matching with General Discrete Paths: A Kinetic-Optimal Perspective](https://mlsanity.com/paper/2412.03487/) (2024)
- [Fine-Tuning Discrete Diffusion Models with Policy Gradient Methods](https://mlsanity.com/paper/2502.01384/) (2025)
- [Improving Reasoning for Diffusion Language Models via Group Diffusion Policy Optimization](https://mlsanity.com/paper/2510.08554/) (2025)
- [A Theoretical Analysis of Discrete Flow Matching Generative Models](https://mlsanity.com/paper/2509.22623/) (2025)
- [MaskFlow: Discrete Flows For Flexible and Efficient Long Video Generation](https://mlsanity.com/paper/2502.11234/) (2025)
- [Uniform Discrete Diffusion with Metric Path for Video Generation](https://mlsanity.com/paper/2510.24717/) (2025)
- [All-atom inverse protein folding through discrete flow matching](https://mlsanity.com/paper/2507.14156/) (2025)
- [Regulatory DNA sequence Design with Reinforcement Learning](https://mlsanity.com/paper/2503.07981/) (2025)
- [Flow Matching for Offline Reinforcement Learning with Discrete Actions](https://mlsanity.com/paper/2602.06138/) (2026)
- [Drax: Speech Recognition with Discrete Flow Matching](https://mlsanity.com/paper/2510.04162/) (2025)

## Built on by (most cited first, 3 of 3)

- [RNA Design via Conditioned Flow Matching and Finite-Policy Reinforcement Learning](https://mlsanity.com/paper/2609.36885/) (2026)
- [Reinforcement Learning on the Discrete Composition Channel of a Crystal Generator: Validated Gains and Reward Hacking](https://mlsanity.com/paper/2610.03880/) (2026)
- [D-DOIT: Training-free Adaptation of Discrete Diffusion via Doob's h-Transform](https://mlsanity.com/paper/2610.04938/) (2026)

## Abstract (arXiv)

We introduce Discrete flow Matching policy Optimization (DoMinO), a unified framework for Reinforcement Learning (RL) fine-tuning Discrete Flow Matching (DFM) models under a broad class of policy gradient methods. Our key idea is to view the DFM sampling procedure as a multi-step Markov Decision Process. This perspective provides a simple and transparent reformulation of fine-tuning reward maximization as a robust RL objective. Consequently, it not only preserves the original DFM samplers but also avoids biased auxiliary estimators and likelihood surrogates used by many prior RL fine-tuning methods. To prevent policy collapse, we also introduce new total-variation regularizers to keep the fine-tuned distribution close to the pretrained one. Theoretically, we establish an upper bound on the discretization error of DoMinO and tractable upper bounds for the regularizers. Experimentally, we evaluate DoMinO on regulatory DNA sequence design. DoMinO achieves stronger predicted enhancer activity and better sequence naturalness than the previous best reward-driven baselines. The regularization further improves alignment with the natural sequence distribution while preserving strong functional performance. These results establish DoMinO as an useful framework for controllable discrete sequence generation.