mlsanityData to 7 Oct 2026

Diffusion-State Policy Optimization for Masked Diffusion Language Models

arXiv 2602.06462 ↗First posted 6 Feb 2026cs.CL cs.LGcited by 3

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Masked diffusion language models generate text through iterative masked-token filling, but terminal-only rewards on final completions provide coarse credit assignment for the intermediate filling decisions that shape the generation process. We propose Diffusion-State Policy Optimization (DiSPO), a plug-in credit-assignment layer that directly optimizes intermediate filling decisions. At selected intermediate masked states, DiSPO branches by resampling the currently masked positions from rollout-cached logits, scores the resulting completions, and updates only the newly filled tokens, requiring no additional multi-step diffusion rollouts or optimizer steps. We formalize a fixed-state objective for branched completions and derive a policy-gradient estimator that reuses the same rollouts as terminal-feedback policy optimization. Experiments on LLaDA-8B-Instruct show that DiSPO consistently improves terminal-feedback baselines, including diffu-GRPO and SPG, on math and planning benchmarks under matched rollout compute and optimizer steps, supporting its use as a general plug-in for masked diffusion policy optimization. Our project page is available at https://daioba.github.io/dispo .

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 30 of its 33 references that are in the map. First posted on arXiv 6 Feb 2026.

Builds on

30 of 33 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025SPG: Sandwiched Policy Gradient for Masked Diffusion Language Models10 Oct
  2. 2025Improving Reasoning for Diffusion Language Models via Group Diffusion Policy Optimization9 Oct
  3. 2025DiFFPO: Training Diffusion LLMs to Reason Fast and Furious via Reinforcement Learning2 Oct
  4. 2025Taming Masked Diffusion Language Models via Consistency Trajectory Reinforcement Learning with Fewer Decoding Step28 Sep
  5. 2025d2: Improving Reasoning in Diffusion Language Models via Trajectory Likelihood Estimation25 Sep
  6. 2025Dream 7B: Diffusion Large Language Models21 Aug
  7. 2025MDPO: Overcoming the Training-Inference Divide of Masked Diffusion Language Models18 Aug
  8. 2025wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Models7 Jul
  9. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation25 Jun
  10. 2025Plan for Speed: Dilated Scheduling for Masked Diffusion Language Models23 Jun
  11. 2025dKV-Cache: The Cache for Diffusion Language Models21 May
  12. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models12 Mar
3 more
  1. 2025Large Language Diffusion Models14 Feb
  2. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  3. 2024Simplified and Generalized Masked Diffusion for Discrete Data6 Jun

Other references · 15

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2023Let's Verify Step by Stepcited by 3,598
  6. 2020Learning to summarize from human feedbackcited by 2,562
  7. 2015Trust Region Policy Optimizationcited by 2,503
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  9. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  10. 2025s1: Simple test-time scalingcited by 1,311
  11. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 404
  12. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learningcited by 178
3 more
  1. 2025dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Cachingcited by 168
  2. 2025Accelerating Diffusion LLMs via Adaptive Parallel Decodingcited by 108
  3. 2025Fine-Tuning Discrete Diffusion Models with Policy Gradient Methodscited by 36

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026A Continuous-Time Reinforcement Learning Framework for Fine-Tuning Discrete Diffusion Modelscited by 1
  2. 2026CT-OPD: Counterfactual Trace On-Policy Distillation for Diffusion Vision-Language Modelscited by 0
  3. 2026Exploring More, Reasoning Better: Stepwise Risk-Sensitive GRPO for Diffusion Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Mask-Aware Policy Gradients for Diffusion Language Models
  2. 2026The Path Matters: Learning a Token-Commitment Policy for Diffusion Language Models
  3. 2026From Position Risks to Block Survival: Faster Generation for Diffusion Language Models
  4. 2025Learning Unmasking Policies for Diffusion Language Models
  5. 2026OPTD: On-Policy Transition Distillation with Consistency-Guided Adaptive Compression for Few-Step Diffusion Language Models
  6. 2026Beyond Masks: Efficient, Flexible Diffusion Language Models via Deletion-Insertion Processes
  7. 2025On Powerful Ways to Generate: Autoregression, Diffusion, and Beyond
  8. 2026Parallelism and Generation Order in Masked Diffusion Language Models: Limits Today, Potential Tomorrow