mlsanityData to 14 Oct 2026

Revise, Don't Freeze: Sampler-Matched Training for Self-Correcting Masked Diffusion Language Models

arXiv 2606.01026 ↗First posted 31 May 2026cs.CLcited by 1

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Masked diffusion language models (MDLMs) re-predict every position at each denoising step, but standard samplers commit tokens once revealed, leaving this revision capability unused. Existing approaches either add heuristic or learned mechanisms to revise committed tokens, or remask them back to [MASK] before re-predicting; a principled sampler that directly revises visible tokens without auxiliary modules remains underexplored. We introduce D3IM, a parameter-free sampler derived as a corrector-style reverse update that permits direct visible-to-visible revision without additional modules or auxiliary passes. D3IM also reveals a model-side obstacle we term preservation bias: the model tends to reproduce its own wrong committed tokens rather than correct them. We address this with SCOPE (Self-Conditioned On Prediction Errors), a lightweight post-training procedure that simulates D3IM's sampling process. On LLaDA-8B at 64 denoising steps, SCOPE+D3IM improves over the original LLaDA-8B with standard unmasking by +13.0 on GSM8K (68.3%), +4.8 on MATH-500 (23.6%), +15.3 on HumanEval (29.3%), and +10.4 on MBPP (30.8%), with gains that increase as more denoising steps are used on math and HumanEval.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 14 of its 23 references that are in the map. First posted on arXiv 31 May 2026.

Builds on

14 of 23 references.

Closest ancestors · 8

Same direction, newest first

  1. 2026Early Decisions Matter: Proximity Bias and Initial Trajectory Shaping in Non-Autoregressive Diffusion Language Models12 Apr
  2. 2026DMax: Aggressive Parallel Decoding for dLLMs9 Apr
  3. 2026Learn from Your Mistakes: Self-Correcting Masked Diffusion Models12 Feb
  4. 2026Balancing Understanding and Generation in Discrete Diffusion Models1 Feb
  5. 2025Fine-Tuning Masked Diffusion for Provable Self-Correction1 Oct
  6. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation25 Jun
  7. 2025Remasking Discrete Diffusion Models with Inference-Time Scaling1 Mar
  8. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution25 Oct

Other references · 6

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2023Let's Verify Step by Stepcited by 3,789
  4. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  5. 2025Wide-In, Narrow-Out: Revokable Decoding for Efficient and Effective DLLMscited by 46
  6. 2025Finish First, Perfect Later: Test-Time Token-Level Cross-Validation for Diffusion Large Language Modelscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Answer First, Reason Later: When Commitment Order Costs Accuracy in Diffusion Language Modelscited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2026Remask, Don't Replace: Token-to-Mask Refinement in Diffusion Language Models
  2. 2026Attention-Discounted Adaptive Sampler for Masked Diffusion Language Models
  3. 2026CoDR: Training-Free Confidence-Drift Remasking for Diffusion Language Models
  4. 2026Know When to Hold 'em: Correct-Token Retention in Uniform-State Diffusion Language Models
  5. 2026CORE: Context-Robust Remasking for Diffusion Language Models
  6. 2026Not All Denoising Steps Are Equal: Model Scheduling for Faster Masked Diffusion Language Models
  7. 2026LESS Is More: Mutual-Stability Sampling for Diffusion Language Models
  8. 2026Re-evaluating Confidence Remasking in Masked Diffusion Language Models