mlsanityData to 14 Oct 2026

FlowRL: Matching Reward Distributions for LLM Reasoning

arXiv 2509.15207 ↗First posted 18 Sep 2025cs.LG cs.AI cs.CLcited by 40

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

We propose FlowRL: matching the full reward distribution via flow balancing instead of maximizing rewards in large language model (LLM) reinforcement learning (RL). Recent advanced reasoning models adopt reward-maximizing methods (eg, PPO and GRPO), which tend to over-optimize dominant reward signals while neglecting less frequent but valid reasoning paths, thus reducing diversity. In contrast, we transform scalar rewards into a normalized target distribution using a learnable partition function, and then minimize the reverse KL divergence between the policy and the target distribution. We implement this idea as a flow-balanced optimization method that promotes diverse exploration and generalizable reasoning trajectories. We conduct experiments on math and code reasoning tasks: FlowRL achieves a significant average improvement of 10.0% over GRPO and 5.1% over PPO on math benchmarks, and performs consistently better on code reasoning tasks. These results highlight reward distribution-matching as a key step toward efficient exploration and diverse reasoning in LLM reinforcement learning.

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 67 of its 83 references that are in the map. First posted on arXiv 18 Sep 2025.

Builds on

67 of 83 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Models10 Sep
  2. 2025Agentic Reinforced Policy Optimization26 Jul
  3. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
  4. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  5. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  6. 2025SRPO: A Cross-Domain Implementation of Large-Scale Reinforcement Learning on LLM19 Apr
  7. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  8. 2025Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training24 Mar
  9. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  10. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  11. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  12. 2023Let's Verify Step by Step31 May
1 more
  1. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 54

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2022Flow Matching for Generative Modelingcited by 5,702
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  6. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  7. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  8. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  9. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  10. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  11. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  12. 2025Group Sequence Policy Optimizationcited by 835
38 more
  1. 2021Flow Network based Generative Models for Non-Iterative Diverse Candidate Generationcited by 392
  2. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
  3. 2021GFlowNet Foundationscited by 290
  4. 2022Trajectory balance: Improved credit assignment in GFlowNetscited by 251
  5. 2024Reinforcement learningcited by 247
  6. 2025Flow Q-Learningcited by 184
  7. 2018Understanding the impact of entropy on policy optimizationcited by 176
  8. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 176
  9. 2022Biological Sequence Design with GFlowNetscited by 174
  10. 2022Learning GFlowNets from partial episodes for improved convergence and stabilitycited by 132
  11. 2022Generative Flow Networks for Discrete Probabilistic Modelingcited by 114
  12. 2022GFlowNets and variational inferencecited by 106
  13. 2022Multi-Objective GFlowNetscited by 101
  14. 2025Flow Matching Policy Gradientscited by 99
  15. 2023Amortizing intractable inference in large language modelscited by 97
  16. 2023Better Training of GFlowNets with Local Credit and Incomplete Trajectoriescited by 85
  17. 2023Towards Understanding and Improving GFlowNet Trainingcited by 75
  18. 2023Robust Scheduling with GFlowNetscited by 69
  19. 2023Generative Flow Networks as Entropy-Regularized RLcited by 64
  20. 2023GFlowNets for AI-Driven Scientific Discoverycited by 63
  21. 2023Diffusion Generative Flow Samplers: Improving learning signals through partial trajectory optimizationcited by 57
  22. 2024Learning diverse attacks on large language models for robust red-teaming and safety tuningcited by 54
  23. 2022Generative Augmented Flow Networkscited by 50
  24. 2024Discrete Probabilistic Inference as Control in Multi-path Environmentscited by 44
  25. 2023Let the Flows Tell: Solving Graph Combinatorial Optimization Problems with GFlowNetscited by 40
  26. 2022A Variational Perspective on Generative Flow Networkscited by 39
  27. 2025Flow-Based Policy for Online Reinforcement Learningcited by 37
  28. 2023Stochastic Generative Flow Networkscited by 32
  29. 2024Efficient Diversity-Preserving Diffusion Alignment via Gradient-Informed GFlowNetscited by 32
  30. 2023Maximum entropy GFlowNets with soft Q-learningcited by 30
  31. 2024Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examplescited by 30
  32. 2023Distributional GFlowNets with Quantile Flowscited by 28
  33. 2023Learning to Scale Logits for Temperature-Conditional GFlowNetscited by 27
  34. 2024Improving GFlowNets for Text-to-Image Diffusion Alignmentcited by 27
  35. 2024SynFlowNet: Design of Diverse and Novel Molecules with Synthesis Constraintscited by 26
  36. 2024Maximum Entropy Reinforcement Learning via Energy-Based Normalizing Flowcited by 26
  37. 2022GFlowOut: Dropout with Generative Flow Networkscited by 22
  38. 2023Pre-Training and Fine-Tuning Generative Flow Networkscited by 17

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 35.

40 papers.

Sort:

By area:

  1. 2025BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clippingcited by 47
  2. 2025When Importance Sampling Misallocates Credit: Asymmetric Ratios for Outcome-Supervised RLcited by 21
  3. 2025Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimizationcited by 20
  4. 2026OGLS-SD: On-Policy Self-Distillation with Outcome-Guided Logit Steering for LLM Reasoningcited by 16
  5. 2026HapticVLA: Contact-Rich Manipulation via Vision-Language-Action Model without Inference-Time Tactile Sensingcited by 10
  6. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 7
  7. 2026ResRL: Boosting LLM Reasoning via Negative Sample Projection Residual Reinforcement Learningcited by 6
  8. 2026Beyond Mode Collapse: Distribution Matching for Diverse Reasoningcited by 6
  9. 2026Expected Return Causes Outcome-Level Mode Collapse in Reinforcement Learning and How to Fix It with Inverse Probability Scalingcited by 5
  10. 2026Diffusion Reinforcement Learning via Centered Reward Distillationcited by 5
  11. 2026Rewards as Labels: Revisiting RLVR from a Classification Perspectivecited by 4
  12. 2026TMPO: Trajectory Matching Policy Optimization for Diverse and Efficient Diffusion Alignmentcited by 4
28 more
  1. 2026Matching Features, Not Tokens: Energy-Based Fine-Tuning of Language Modelscited by 3
  2. 2026Listwise Policy Optimization: Group-based RLVR as Target-Projection on the LLM Response Simplexcited by 3
  3. 2026Beyond Mode-Seeking RL: Trajectory-Balance Post-Training for Diffusion Language Modelscited by 3
  4. 2026Latent Thought Flow: Efficient Latent Reasoning in Large Language Modelscited by 3
  5. 2026Efficient Paths and Dense Rewards: Probabilistic Flow Reasoning for Large Language Modelscited by 2
  6. 2026Back to Basics: Revisiting Exploration in Reinforcement Learning for LLM Reasoning via Generative Probabilitiescited by 2
  7. 2026LAD: Learning Advantage Distribution for Reasoningcited by 2
  8. 2026PowerFlow: Unlocking the Dual Nature of LLMs via Principled Distribution Matchingcited by 2
  9. 2026GFlowRL: Scaling Distribution-Matching RL to Large Language Modelscited by 2
  10. 2026Diffusion Alignment Beyond KL: Variance Minimisation as Effective Policy Optimisercited by 1
  11. 2026Beyond Normalization: Rethinking the Partition Function as a Difficulty Scheduler for RLVRcited by 1
  12. 2026Does LLM Alignment Really Need Diversity? An Empirical Study of Adapting RLVR Methods for Moral Reasoningcited by 1
  13. 2026ScoRe-Flow: Complete Distributional Control via Score-Based Reinforcement Learning for Flow Matchingcited by 1
  14. 2026The Scaling Properties of Implicit Deductive Reasoning in Transformerscited by 1
  15. 2026Cyclical Entropy Eruption: Entropy Dynamics in Agent Reinforcement Learningcited by 1
  16. 2026Beyond the Mean: Multi-Moment Policy Optimization for LLM Reasoningcited by 1
  17. 2026FluxLite: Inference-Time Proposal Control for Discrete Diffusion Modelscited by 1
  18. 2025Enhancing LLM Reasoning via Non-Human-Like Reasoning Path Preference Optimizationcited by 0
  19. 2025From generative AI to the brain: five takeawayscited by 0
  20. 2026Controlling Exploration-Exploitation in GFlowNets via Markov Chain Perspectivescited by 0
  21. 2026DFPO: Scaling Value Modeling via Distributional Flow towards Robust and Generalizable LLM Post-Trainingcited by 0
  22. 2026Distributionally Robust Token Optimization in RLHFcited by 0
  23. 2026DISA: Offline Importance Sampling for Distribution-Matching LLM-RLcited by 0
  24. 2026Your GFlowNet Secretly Learns an Optimal Transport Plancited by 0
  25. 2026Unified Trajectory Matching Policy Optimization: Diverse T2I Generation and VLA Generalizationcited by 0
  26. 2026Sample What You Say: Aligning Language Models to Sample the Distributions They Statecited by 0
  27. 2026Beyond Policy Rollouts: Embedding-Space Directional Exploration with Anchored Noise for LLM RLVRcited by 0
  28. 2026RL Post-Training Diffusion Language Models Along Their Own Generation Pathscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Likelihood-Based Reward Designs for General LLM Reasoning
  2. 2025Revisiting LLM Reasoning via Information Bottleneck
  3. 2026Gradients Know What Outcomes Don't: Unlocking Reinforcement Learning for LLM Reasoning with Gradient-Aligned Rewards
  4. 2025Unlocking Reasoning Capabilities in LLMs via Reinforcement Learning Exploration
  5. 2025Count Counts: Motivating Exploration in LLM Reasoning with Count-based Intrinsic Rewards
  6. 2026SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoning
  7. 2026The Art of Efficient Reasoning: Data, Reward, and Optimization
  8. 2025Rectifying LLM Thought from Lens of Optimization