mlsanityData to 7 Oct 2026

SPG: Sandwiched Policy Gradient for Masked Diffusion Language Models

arXiv 2510.09541 ↗First posted 10 Oct 2025cs.CL cs.AIcited by 38

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Diffusion large language models (dLLMs) are emerging as an efficient alternative to autoregressive models due to their ability to decode multiple tokens in parallel. However, aligning dLLMs with human preferences or task-specific rewards via reinforcement learning (RL) is challenging because their intractable log-likelihood precludes the direct application of standard policy gradient methods. While prior work uses surrogates like the evidence lower bound (ELBO), these one-sided approximations can introduce significant policy gradient bias. To address this, we propose the Sandwiched Policy Gradient (SPG) that leverages both an upper and a lower bound of the true log-likelihood. Experiments show that SPG significantly outperforms baselines based on ELBO or one-step estimation. Specifically, SPG improves the accuracy over state-of-the-art RL methods for dLLMs by 3.6% in GSM8K, 2.6% in MATH500, 18.4% in Countdown and 27.0% in Sudoku.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 48 of its 52 references that are in the map. First posted on arXiv 10 Oct 2025.

Builds on

48 of 52 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Models8 Sep
  2. 2025Blockwise SFT for Diffusion Language Models: Reconciling Bidirectional Attention and Autoregressive Decoding27 Aug
  3. 2025Diffusion LLMs Can Do Faster-Than-AR Inference via Discrete Diffusion Forcing8 Aug
  4. 2025wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Models7 Jul
  5. 2025The Diffusion Duality12 Jun
  6. 2025dKV-Cache: The Cache for Diffusion Language Models21 May
  7. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models12 Mar
  8. 2025Large Language Diffusion Models14 Feb
  9. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Models23 Oct
  10. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  11. 2024Simplified and Generalized Masked Diffusion for Discrete Data6 Jun
  12. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution25 Oct
3 more
  1. 2023A Reparameterized Discrete Diffusion Model for Text Generation11 Feb
  2. 2022SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Control31 Oct
  3. 2022DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models17 Oct

Other references · 33

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  6. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  8. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  9. 2022Scalable Diffusion Models with Transformerscited by 5,931
  10. 2017Deep reinforcement learning from human preferencescited by 3,846
  11. 2023Let's Verify Step by Stepcited by 3,598
  12. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
21 more
  1. 2015Trust Region Policy Optimizationcited by 2,503
  2. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  3. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  4. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  5. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  6. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  7. 2025Group Sequence Policy Optimizationcited by 788
  8. 2022A Continuous Time Framework for Discrete Denoising Modelscited by 437
  9. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 431
  10. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 431
  11. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 404
  12. 2024Generative Flows on Discrete State-Spaces: Enabling Multimodal Flows with Applications to Protein Co-Designcited by 294
  13. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 273
  14. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 241
  15. 2025Mercury: Ultra-Fast Language Models Based on Diffusioncited by 225
  16. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learningcited by 178
  17. 2025dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Cachingcited by 168
  18. 2022Score-based Continuous-time Discrete Diffusion Modelscited by 167
  19. 2024Fine-Tuning Discrete Diffusion Models via Reward Optimization with Applications to DNA and Protein Designcited by 72
  20. 2025Esoteric Language Models: A Family of Any-Order Diffusion LLMscited by 33
  21. 2025Soft Policy Optimization: Online Off-Policy RL for Sequence Modelscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 34.

38 papers.

Sort:

By area:

  1. 2025LLaDA2.0: Scaling Up Diffusion Language Models to 100Bcited by 154
  2. 2026LLaDA2.1: Speeding Up Text Diffusion via Token Editingcited by 48
  3. 2025Learning Unmasking Policies for Diffusion Language Modelscited by 29
  4. 2025dUltra: Ultra-Fast Diffusion Language Models via Reinforcement Learningcited by 10
  5. 2026Stabilizing Reinforcement Learning for Diffusion Language Modelscited by 9
  6. 2026ρ-textttEOS: Training-free Bidirectional Variable-Length Control for Masked Diffusion LLMscited by 7
  7. 2026LaDi-RL: Latent Diffusion Reasoning Prevents Entropy Collapse in Reinforcement Learningcited by 7
  8. 2026DARE: Diffusion Large Language Models Alignment and Reinforcement Executorcited by 5
  9. 2026A Tale of Two Temperatures: Simple, Efficient, and Diverse Sampling from Diffusion Language Modelscited by 5
  10. 2026GDSD: Reinforcement Learning as Guided Denoiser Self-Distillation for Diffusion Language Modelscited by 5
  11. 2026Early Decisions Matter: Proximity Bias and Initial Trajectory Shaping in Non-Autoregressive Diffusion Language Modelscited by 4
  12. 2026DreamReasoner-8B: Block-Size Curriculum Learning for Diffusion Reasoning Modelscited by 4
26 more
  1. 2026Diffusion-State Policy Optimization for Masked Diffusion Language Modelscited by 3
  2. 2026Reinforcement Learning for Diffusion LLMs with Entropy-Guided Step Selection and Stepwise Advantagescited by 3
  3. 2026dTRPO: Trajectory Reduction in Policy Optimization of Diffusion Large Language Modelscited by 3
  4. 2026ReflectDrive-2: Reinforcement-Learning-Aligned Self-Editing for Discrete Diffusion Drivingcited by 3
  5. 2025Reasoning in Diffusion Large Language Models is Concentrated in Dynamic Confusion Zonescited by 2
  6. 2026Efficient and Stable Reinforcement Learning for Diffusion Language Modelscited by 2
  7. 2026LFPO: Likelihood-Free Policy Optimization for Masked Diffusion Modelscited by 2
  8. 2026Reasoning on the Manifold: Bidirectional Consistency for Self-Verification in Diffusion Language Modelscited by 2
  9. 2026Beyond Mode-Seeking RL: Trajectory-Balance Post-Training for Diffusion Language Modelscited by 2
  10. 2026NaRA: Noise-Aware LoRA for Parameter-Efficient Fine-Tuning of Diffusion LLMscited by 2
  11. 2026Mask-Aware Policy Gradients for Diffusion Language Modelscited by 2
  12. 2026Lookahead Path Likelihood Optimization for Diffusion LLMscited by 1
  13. 2026TRIMS: Trajectory-Ranked Instruction Masked Supervision for Diffusion Language Modelscited by 1
  14. 2026TUBE: Tangent Upper Bound on Evidence for Discrete Diffusion Language Modelscited by 1
  15. 2026Scheduling Thoughts: Learning the Order of Thought in Diffusion Language Modelscited by 1
  16. 2026Accelerating Masked Diffusion Large Language Models: A Survey of Efficient Inference Techniquescited by 1
  17. 2026A Continuous-Time Reinforcement Learning Framework for Fine-Tuning Discrete Diffusion Modelscited by 1
  18. 2026Progressive Refinement Regulation for Accelerating Diffusion Language Model Decodingcited by 0
  19. 2026Discrete Tilt Matchingcited by 0
  20. 2026Relative Score Policy Optimization for Diffusion Language Modelscited by 0
  21. 2026DiPOD: Diffusion Policy Optimization without Drifting Apartcited by 0
  22. 2026Beyond On-Policy Exploration: Integrating External Policy Rollouts for Reinforcement Learning in Diffusion Language Modelscited by 0
  23. 2026Hesitation-Aware On-Policy Distillation for Diffusion Language Modelscited by 0
  24. 2026Low-Confidence Remasking Traps Flexibility: Realizing Arbitrary-Order Potential for Diverse Rollouts in Diffusion LLMscited by 0
  25. 2026Fork-dLLM: Avoiding the Flexibility Trap in Diffusion Language Modelscited by 0
  26. 2026Exploring More, Reasoning Better: Stepwise Risk-Sensitive GRPO for Diffusion Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Simple Policy Gradients for Reasoning with Diffusion Language Models
  2. 2025Improving Reasoning for Diffusion Language Models via Group Diffusion Policy Optimization
  3. 2025DiRL: An Efficient Post-Training Framework for Diffusion Language Models
  4. 2025MDPO: Overcoming the Training-Inference Divide of Masked Diffusion Language Models
  5. 2025d2: Improving Reasoning in Diffusion Language Models via Trajectory Likelihood Estimation
  6. 2025RFG: Test-Time Scaling for Diffusion Large Language Model Reasoning with Reward-Free Guidance
  7. 2025Principled RL for Diffusion LLMs Emerges from a Sequence-Level Perspective
  8. 2025Advantage Weighted Matching: Aligning RL with Pretraining in Diffusion Models