# Graph-GRPO: Training Graph Flow Models with Reinforcement Learning

arXiv: https://arxiv.org/abs/2603.10395 · HTML: https://mlsanity.com/paper/2603.10395/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 38 of its 52 references that are in the map.
- First posted on arXiv 11 Mar 2026.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 30 of 38)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Structured Denoising Diffusion Models in Discrete State-Spaces](https://mlsanity.com/paper/2107.03006/) (2021)
- [Training Diffusion Models with Reinforcement Learning](https://mlsanity.com/paper/2305.13301/) (2023)
- [Junction Tree Variational Autoencoder for Molecular Graph Generation](https://mlsanity.com/paper/1802.04364/) (2018)
- [DiGress: Discrete Denoising diffusion for graph generation](https://mlsanity.com/paper/2209.14734/) (2022)
- [A Continuous Time Framework for Discrete Denoising Models](https://mlsanity.com/paper/2205.14987/) (2022)
- [GraphRNN: Generating Realistic Graphs with Deep Auto-regressive Models](https://mlsanity.com/paper/1802.08773/) (2018)
- [MolGAN: An implicit generative model for small molecular graphs](https://mlsanity.com/paper/1805.11973/) (2018)
- [Graph Convolutional Policy Network for Goal-Directed Molecular Graph Generation](https://mlsanity.com/paper/1806.02473/) (2018)
- [Discrete Flow Matching](https://mlsanity.com/paper/2407.15595/) (2024)
- [Generative Flows on Discrete State-Spaces: Enabling Multimodal Flows with Applications to Protein Co-Design](https://mlsanity.com/paper/2402.04997/) (2024)
- [Score-based Generative Modeling of Graphs via the System of Stochastic Differential Equations](https://mlsanity.com/paper/2202.02514/) (2022)
- [Molecular De Novo Design through Deep Reinforcement Learning](https://mlsanity.com/paper/1704.07555/) (2017)
- [Sample Efficiency Matters: A Benchmark for Practical Molecular Optimization](https://mlsanity.com/paper/2206.12411/) (2022)
- [Graph Inductive Biases in Transformers without Message Passing](https://mlsanity.com/paper/2305.17589/) (2023)
- [SPECTRE: Spectral Conditioning Helps to Overcome the Expressivity Limits of One-shot Graph Generators](https://mlsanity.com/paper/2204.01613/) (2022)
- [Exploring Chemical Space with Score-based Out-of-distribution Generation](https://mlsanity.com/paper/2206.07632/) (2022)
- [Variational Flow Matching for Graph Generation](https://mlsanity.com/paper/2406.04843/) (2024)
- [DeFoG: Discrete Flow Matching for Graph Generation](https://mlsanity.com/paper/2410.04263/) (2024)
- [Discrete-state Continuous-time Diffusion for Graph Generation](https://mlsanity.com/paper/2405.11416/) (2024)
- [Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers](https://mlsanity.com/paper/2510.11370/) (2025)
- [OntoProtein: Protein Pretraining With Gene Ontology Embedding](https://mlsanity.com/paper/2201.11147/) (2022)
- [Hit and Lead Discovery with Explorative RL and Fragment-based Molecule Generation](https://mlsanity.com/paper/2110.01219/) (2021)
- [GraphGen: A Scalable Approach to Domain-agnostic Labeled Graph Generation](https://mlsanity.com/paper/2001.08184/) (2020)
- [Scalable Deep Generative Modeling for Sparse Graphs](https://mlsanity.com/paper/2006.15502/) (2020)
- [GenMol: A Drug Discovery Generalist with Discrete Diffusion](https://mlsanity.com/paper/2501.06158/) (2025)
- [Genetic-guided GFlowNets for Sample Efficient Molecular Optimization](https://mlsanity.com/paper/2402.05961/) (2024)
- [Chemical-Reaction-Aware Molecule Representation Learning](https://mlsanity.com/paper/2109.09888/) (2021)

## Built on by (most cited first, 5 of 5)

- [Crys-JEPA: Accelerating Crystal Discovery via Embedding Screening and Generative Refinement](https://mlsanity.com/paper/2605.14759/) (2026)
- [Composable Crystals: Controllable Materials Discovery via Concept Learning](https://mlsanity.com/paper/2605.14769/) (2026)
- [Reinforcement Learning on the Discrete Composition Channel of a Crystal Generator: Validated Gains and Reward Hacking](https://mlsanity.com/paper/2610.03880/) (2026)
- [STAMP as a topology-aware molecular language for fragment-level learning and editing](https://mlsanity.com/paper/2610.18438/) (2026)
- [Learning Graph Energies for Finite-Time Generation](https://mlsanity.com/paper/2610.19946/) (2026)

## Abstract (arXiv)

Graph generation is a fundamental task with broad applications, such as drug discovery. Recently, discrete flow matching-based graph generation, aka, graph flow model (GFM), has emerged due to its superior performance and flexible sampling. However, effectively aligning GFMs with complex human preferences or task-specific objectives remains a significant challenge. In this paper, we propose Graph-GRPO, an online reinforcement learning (RL) framework for training GFMs under verifiable rewards. Our method makes two key contributions: (1) We derive an analytical expression for the transition probability of GFMs, replacing the Monte Carlo sampling and enabling fully differentiable rollouts for RL training; (2) We propose a refinement strategy that randomly perturbs specific nodes and edges in a graph, and regenerates them, allowing for localized exploration and self-improvement of generation quality. Extensive experiments on both synthetic and real datasets demonstrate the effectiveness of Graph-GRPO. With only 50 denoising steps, our method achieves 95.0% and 97.5% Valid-Unique-Novelty scores on the planar and tree datasets, respectively. Moreover, Graph-GRPO achieves state-of-the-art performance on the molecular optimization tasks, outperforming graph-based and fragment-based RL methods as well as classic genetic algorithms.