mlsanityData to 7 Oct 2026

Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimization

arXiv 2510.08233 ↗First posted 9 Oct 2025cs.LGcited by 15

On the map: LLM reasoning → Diffusion reasoning and test-time scaling

Abstract (arXiv)

Diffusion large language models (dLLMs) are promising alternatives to autoregressive large language models (AR-LLMs), as they potentially allow higher inference throughput. Reinforcement learning (RL) is crucial to enabling dLLMs to achieve performance comparable to that of AR-LLMs on important tasks, such as reasoning. However, RL algorithms well-suited to dLLMs' unique characteristics have yet to be developed. This paper proposes Distribution Matching Policy Optimization (DMPO), a principled and theoretically grounded RL fine-tuning method specifically designed to enhance the reasoning capabilities of dLLMs by matching the dLLM policy distribution to the optimal, reward-tilted one through cross-entropy optimization. We identify a key implementation challenge with small training batch sizes and propose several effective solutions based on a novel weight baseline subtraction technique. DMPO exhibits superior performance on multiple reasoning benchmarks without supervised fine-tuning, achieving up to a 39.63 percentage-point improvement in accuracy over prior non-DMPO RL baselines and 67.97 percentage points over the base model, underscoring the effectiveness of the distribution-matching framework. Our code is available at https://github.com/yuchen-zhu-zyc/DMPO.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Diffusion reasoning and test-time scaling, one of 17 directions in LLM reasoning. Builds on 90 of its 103 references that are in the map. First posted on arXiv 9 Oct 2025.

Builds on

90 of 103 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Coevolutionary Continuous Discrete Diffusion: Make Your Diffusion Language Model a Latent Reasoner3 Oct
  2. 2025DriftLite: Lightweight Drift Control for Inference-Time Scaling of Diffusion Models25 Sep
  3. 2025wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Models7 Jul
  4. 2025Mercury: Ultra-Fast Language Models Based on Diffusion17 Jun
  5. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning16 Apr

Other references · 85

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  6. 2022Generative Adversarial Networkscited by 5,603
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  9. 2023Let's Verify Step by Stepcited by 3,598
  10. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  11. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  12. 2015Trust Region Policy Optimizationcited by 2,503
38 more
  1. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  2. 2025s1: Simple test-time scalingcited by 1,311
  3. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  4. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  5. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  6. 2025Large Language Diffusion Modelscited by 872
  7. 2024Simple and Effective Masked Diffusion Language Modelscited by 800
  8. 2025Group Sequence Policy Optimizationcited by 788
  9. 2025AlphaEvolve: A coding agent for scientific and algorithmic discoverycited by 763
  10. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 712
  11. 2024Simplified and Generalized Masked Diffusion for Discrete Datacited by 606
  12. 2025Flow-GRPO: Training Flow Matching Models via Online RLcited by 605
  13. 2025Dream 7B: Diffusion Large Language Modelscited by 515
  14. 2022A Continuous Time Framework for Discrete Denoising Modelscited by 437
  15. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 404
  16. 2025DanceGRPO: Unleashing GRPO on Visual Generationcited by 388
  17. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Modelscited by 385
  18. 2021Flow Network based Generative Models for Non-Iterative Diverse Candidate Generationcited by 384
  19. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Datacited by 371
  20. 2024Generative Flows on Discrete State-Spaces: Enabling Multimodal Flows with Applications to Protein Co-Designcited by 294
  21. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 273
  22. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 241
  23. 2024Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Controlcited by 222
  24. 2025Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusionscited by 208
  25. 2024Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Samplingcited by 204
  26. 2025DiffusionNFT: Online Diffusion Reinforcement with Forward Processcited by 204
  27. 2024Scaling up Masked Diffusion Models on Textcited by 195
  28. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generationcited by 184
  29. 2025Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inferencecited by 173
  30. 2025dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Cachingcited by 168
  31. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoningcited by 147
  32. 2025dKV-Cache: The Cache for Diffusion Language Modelscited by 140
  33. 2025Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmaskingcited by 128
  34. 2024Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decodingcited by 127
  35. 2024Unlocking Guidance for Discrete State-Space Diffusion and Flow Modelscited by 124
  36. 2025Geometric-Mean Policy Optimizationcited by 107
  37. 2016Neural Autoregressive Distribution Estimationcited by 94
  38. 2024Learning Dynamics of LLM Finetuningcited by 93

50 most cited of 85 All 85 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 13.

15 papers.

Sort:

By area:

  1. 2025Learning Unmasking Policies for Diffusion Language Modelscited by 29
  2. 2025dUltra: Ultra-Fast Diffusion Language Models via Reinforcement Learningcited by 10
  3. 2026LaViDa-R1: Advancing Reasoning for Unified Multimodal Diffusion Language Modelscited by 6
  4. 2026GDSD: Reinforcement Learning as Guided Denoiser Self-Distillation for Diffusion Language Modelscited by 5
  5. 2026Discrete Adjoint Schr"odinger Bridge Samplercited by 4
  6. 2026Diffusion Reinforcement Learning via Centered Reward Distillationcited by 3
  7. 2026Contrastive Distribution Matching for Amortized Sequential Monte Carlo in Discrete Diffusioncited by 3
  8. 2026Beyond Mode-Seeking RL: Trajectory-Balance Post-Training for Diffusion Language Modelscited by 2
  9. 2026A2D2: Fine-Tuning Any-Length Discrete Diffusion for Adaptive Decodingcited by 2
  10. 2026DPRM: A Plug-in Doob h transform-induced Token-Ordering Module for Discrete Diffusion Modelscited by 1
  11. 2026Scheduling Thoughts: Learning the Order of Thought in Diffusion Language Modelscited by 1
  12. 2026Back on Track: Aligning Rewards and States for Reasoning in Diffusion Large Language Modelscited by 0
3 more
  1. 2026Graph Forward Distribution Matching for Molecular Inverse Designcited by 0
  2. 2026CT-OPD: Counterfactual Trace On-Policy Distillation for Diffusion Vision-Language Modelscited by 0
  3. 2026Exploring More, Reasoning Better: Stepwise Risk-Sensitive GRPO for Diffusion Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025d-TreeRPO: Towards More Reliable Policy Optimization for Diffusion Language Models
  2. 2025Reasoning in Diffusion Large Language Models is Concentrated in Dynamic Confusion Zones
  3. 2025Simple Policy Gradients for Reasoning with Diffusion Language Models
  4. 2026Break the Block: Dynamic-size Reasoning Blocks for Diffusion Large Language Models via Monotonic Entropy Descent with Reinforcement Learning
  5. 2025DiFFPO: Training Diffusion LLMs to Reason Fast and Furious via Reinforcement Learning
  6. 2026Relative Score Policy Optimization for Diffusion Language Models
  7. 2026The Flexibility Trap: Rethinking the Value of Arbitrary Order in Diffusion Language Models
  8. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regression