mlsanityData to 14 Oct 2026

wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Models

arXiv 2507.08838 ↗First posted 7 Jul 2025cs.LG cs.AI stat.MLcited by 71

On the map: LLM reasoning → Diffusion reasoning and test-time scaling

also in 1 more
Abstract (arXiv)

Improving the reasoning capabilities of diffusion-based large language models (dLLMs) through reinforcement learning (RL) remains an open problem. The intractability of dLLMs likelihood function necessitates approximating the current, old, and reference policy likelihoods at each policy optimization step. This reliance introduces additional computational overhead, and can lead to large variance and estimation error in RL objective -- particularly in computing the policy ratio for importance sampling. To mitigate these issues, we introduce wd1, a novel ratio-free policy optimization approach that reformulates the RL objective as a weighted log-likelihood, requiring only a single approximation for the current parametrized policy likelihood. We formally show that our proposed method can be interpreted as energy-guided discrete diffusion training combined with negative sample unlearning, thereby confirming its theoretical soundness. In experiments on LLaDA-8B model, wd1 outperforms diffusion-based GRPO (d1) while requiring lower computational cost, achieving up to a +59% improvement in accuracy. Furthermore, we extend wd1 to denoising-stepwise weighted policy optimization (wd1++), achieving state-of-the-art math performance of 44.2% on MATH500 and 84.5% on GSM8K with only 20 RL training steps.

Key facts. Cited by 71 papers in the map (data through 14 Oct 2026). Placed in Diffusion reasoning and test-time scaling, one of 17 directions in LLM reasoning. Builds on 48 of its 67 references that are in the map. First posted on arXiv 7 Jul 2025.

Builds on

48 of 67 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Mercury: Ultra-Fast Language Models Based on Diffusion17 Jun
  2. 2025Reinforcing the Diffusion Chain of Lateral Thought with Diffusion Language Models15 May
  3. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning16 Apr
  4. 2025Remasking Discrete Diffusion Models with Inference-Time Scaling1 Mar

Other references · 44

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  7. 2023Let's Verify Step by Stepcited by 3,789
  8. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  9. 2015Trust Region Policy Optimizationcited by 2,580
  10. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  11. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  12. 2025s1: Simple test-time scalingcited by 1,341
32 more
  1. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  2. 2025Large Language Diffusion Modelscited by 912
  3. 2024Simple and Effective Masked Diffusion Language Modelscited by 848
  4. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  5. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 745
  6. 2024Simplified and Generalized Masked Diffusion for Discrete Datacited by 626
  7. 2019Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networkscited by 618
  8. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  9. 2022A Continuous Time Framework for Discrete Denoising Modelscited by 451
  10. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Modelscited by 409
  11. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Datacited by 381
  12. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 285
  13. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 250
  14. 2024Scaling up Masked Diffusion Models on Textcited by 199
  15. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraintscited by 174
  16. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 162
  17. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learningcited by 149
  18. 2022Concrete Score Matching: Generalized Score Matching for Discrete Datacited by 148
  19. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforcecited by 145
  20. 2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimizationcited by 110
  21. 2025Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplificationcited by 110
  22. 2025ACECODER: Acing Coder RL via Automated Test-Case Synthesiscited by 93
  23. 2024Noise Contrastive Alignment of Language Models with Explicit Rewardscited by 85
  24. 2024Fine-Tuning Discrete Diffusion Models via Reward Optimization with Applications to DNA and Protein Designcited by 79
  25. 2025Energy-Weighted Flow Matching for Offline Reinforcement Learningcited by 73
  26. 2023Fine-Tuning Language Models with Advantage-Induced Policy Alignmentcited by 48
  27. 2025Fine-Tuning Discrete Diffusion Models with Policy Gradient Methodscited by 38
  28. 2025NFT: Bridging Supervised Learning and Reinforcement Learning in Math Reasoningcited by 38
  29. 2025Data Unlearning in Diffusion Modelscited by 18
  30. 2023Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Modelscited by 17
  31. 2017f-Divergence constrained policy improvementcited by 16
  32. 2025RLHF in an SFT Way: From Optimal Solution to Reward-Weighted Alignmentcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 26; 2026: 45.

71 papers.

Sort:

By area:

  1. 2025A Survey on Diffusion Language Modelscited by 78
  2. 2025LLaDA-MoE: A Sparse MoE Diffusion Language Modelcited by 57
  3. 2025SPG: Sandwiched Policy Gradient for Masked Diffusion Language Modelscited by 41
  4. 2025Don't Settle Too Early: Self-Reflective Remasking for Diffusion Language Modelscited by 37
  5. 2025Principled RL for Diffusion LLMs Emerges from a Sequence-Level Perspectivecited by 34
  6. 2025Improving Reasoning for Diffusion Language Models via Group Diffusion Policy Optimizationcited by 33
  7. 2025A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Modelscited by 30
  8. 2025Learning Unmasking Policies for Diffusion Language Modelscited by 30
  9. 2025d2: Improving Reasoning in Diffusion Language Models via Trajectory Likelihood Estimationcited by 27
  10. 2025From Next-Token to Next-Block: A Principled Adaptation Path for Diffusion LLMscited by 24
  11. 2025Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimizationcited by 20
  12. 2025DiFFPO: Training Diffusion LLMs to Reason Fast and Furious via Reinforcement Learningcited by 19
38 more
  1. 2026DMax: Aggressive Parallel Decoding for dLLMscited by 18
  2. 2025d-TreeRPO: Towards More Reliable Policy Optimization for Diffusion Language Modelscited by 17
  3. 2025Inpainting-Guided Policy Optimization for Diffusion Large Language Modelscited by 16
  4. 2025Consolidating Reinforcement Learning for Multimodal Discrete Diffusion Modelscited by 12
  5. 2025Simple Policy Gradients for Reasoning with Diffusion Language Modelscited by 12
  6. 2025MAPO: Mixed Advantage Policy Optimizationcited by 11
  7. 2026Breaking the Factorization Barrier in Diffusion Language Modelscited by 11
  8. 2025Advancing Reasoning in Diffusion Language Models with Denoising Process Rewardscited by 10
  9. 2025dUltra: Ultra-Fast Diffusion Language Models via Reinforcement Learningcited by 10
  10. 2026Stabilizing Reinforcement Learning for Diffusion Language Modelscited by 10
  11. 2025Test-Time Scaling in Diffusion LLMs via Hidden Semi-Autoregressive Expertscited by 8
  12. 2025RFG: Test-Time Scaling for Diffusion Large Language Model Reasoning with Reward-Free Guidancecited by 7
  13. 2026LaDi-RL: Latent Diffusion Reasoning Prevents Entropy Collapse in Reinforcement Learningcited by 7
  14. 2026LaViDa-R1: Advancing Reasoning for Unified Multimodal Diffusion Language Modelscited by 7
  15. 2026GDSD: Reinforcement Learning as Guided Denoiser Self-Distillation for Diffusion Language Modelscited by 6
  16. 2026Learning from the Self-future: On-policy Self-distillation for dLLMscited by 5
  17. 2026DreamReasoner-8B: Block-Size Curriculum Learning for Diffusion Reasoning Modelscited by 5
  18. 2025GIFT: Guided Importance-Aware Fine-Tuning for Diffusion Language Modelscited by 4
  19. 2025Beyond Surface Reasoning: Unveiling the True Long Chain-of-Thought Capacity of Diffusion Large Language Modelscited by 4
  20. 2026Diffusion-State Policy Optimization for Masked Diffusion Language Modelscited by 4
  21. 2026Reinforcement Learning for Diffusion LLMs with Entropy-Guided Step Selection and Stepwise Advantagescited by 4
  22. 2026Early Decisions Matter: Proximity Bias and Initial Trajectory Shaping in Non-Autoregressive Diffusion Language Modelscited by 4
  23. 2026NaRA: Noise-Aware LoRA for Parameter-Efficient Fine-Tuning of Diffusion LLMscited by 4
  24. 2026Back on Track: Aligning Rewards and States for Reasoning in Diffusion Large Language Modelscited by 4
  25. 2025Reasoning in Diffusion Large Language Models is Concentrated in Dynamic Confusion Zonescited by 3
  26. 2026dTRPO: Trajectory Reduction in Policy Optimization of Diffusion Large Language Modelscited by 3
  27. 2026Beyond Mode-Seeking RL: Trajectory-Balance Post-Training for Diffusion Language Modelscited by 3
  28. 2026TACG: Trajectory-Aware Commit Gating for Diffusion Language Model Decodingcited by 3
  29. 2026Mask-Aware Policy Gradients for Diffusion Language Modelscited by 3
  30. 2026Efficient and Stable Reinforcement Learning for Diffusion Language Modelscited by 2
  31. 2026Discrete Tilt Matchingcited by 2
  32. 2026dFlowGRPO: Rate-Aware Policy Optimization for Discrete Flow Modelscited by 2
  33. 2026TAD: Temporal-Aware Trajectory Self-Distillation for Fast and Accurate Diffusion LLMcited by 2
  34. 2026dMoE: dLLMs with Learnable Block Expertscited by 2
  35. 2026A Continuous-Time Reinforcement Learning Framework for Fine-Tuning Discrete Diffusion Modelscited by 2
  36. 2025Aligning Diffusion Language Models via Unpaired Preference Optimizationcited by 1
  37. 2025C2DLM: Causal Concept-Guided Diffusion Large Language Modelscited by 1
  38. 2026GeMPO: Generalized Measure Matching for Online Diffusion Reinforcement Learningcited by 1

50 most cited of 71 All 71 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026LFPO: Likelihood-Free Policy Optimization for Masked Diffusion Models
  2. 2025Boundary-Guided Policy Optimization for Memory-efficient RL of Diffusion Large Language Models
  3. 2025Advantage Weighted Matching: Aligning RL with Pretraining in Diffusion Models
  4. 2025DiRL: An Efficient Post-Training Framework for Diffusion Language Models
  5. 2026dOPSD: On-Policy Self-Distillation for Diffusion Language Models
  6. 2026Trace-Based On-Policy Distillation for Masked Diffusion Language Models
  7. 2025MRO: Enhancing Reasoning in Diffusion Language Models via Multi-Reward Optimization
  8. 2025MDPO: Overcoming the Training-Inference Divide of Masked Diffusion Language Models