mlsanityData to 7 Oct 2026

DOS: Dependency-Oriented Sampler for Masked Diffusion Language Models

arXiv 2603.15340 ↗First posted 16 Mar 2026cs.CL stat.MLcited by 2

On the map: Diffusion models → Discrete and text diffusion

also in 1 more
Abstract (arXiv)

Masked diffusion language models (MDLMs) have recently emerged as a new paradigm in language modeling, offering flexible generation dynamics and enabling efficient parallel decoding. However, existing decoding strategies for pre-trained MDLMs predominantly rely on token-level uncertainty criteria, while largely overlooking sequence-level information and inter-token dependencies. To address this limitation, we propose Dependency-Oriented Sampler (DOS), a training-free decoding strategy that leverages inter-token dependencies to inform token updates during generation. Specifically, DOS exploits attention matrices from transformer blocks to approximate inter-token dependencies, emphasizing information from unmasked tokens when updating masked positions. Empirical results demonstrate that DOS consistently achieves superior performance on both code generation and mathematical reasoning tasks. Moreover, DOS can be seamlessly integrated with existing parallel sampling methods, leading to improved generation efficiency without sacrificing generation quality.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 30 of its 43 references that are in the map. First posted on arXiv 16 Mar 2026.

Builds on

30 of 43 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025ReFusion: A Diffusion Large Language Model with Parallel Autoregressive Decoding15 Dec
  2. 2025KLASS: KL-Guided Fast Inference in Masked Diffusion Models7 Nov
  3. 2025Taming Masked Diffusion Language Models via Consistency Trajectory Reinforcement Learning with Fewer Decoding Step28 Sep
  4. 2025Dream 7B: Diffusion Large Language Models21 Aug
  5. 2025Diffusion LLMs Can Do Faster-Than-AR Inference via Discrete Diffusion Forcing8 Aug
  6. 2025Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions10 Feb
  7. 2024Simplified and Generalized Masked Diffusion for Discrete Data6 Jun
  8. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data6 Jun
  9. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution25 Oct

Other references · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  7. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  8. 2023Let's Verify Step by Stepcited by 3,598
  9. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  10. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  11. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  12. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
9 more
  1. 2022A Continuous Time Framework for Discrete Denoising Modelscited by 437
  2. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 404
  3. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decodingcited by 266
  4. 2022Torsional Diffusion for Molecular Conformer Generationcited by 262
  5. 2022Concrete Score Matching: Generalized Score Matching for Discrete Datacited by 144
  6. 2022Directed Acyclic Transformer for Non-Autoregressive Machine Translationcited by 44
  7. 2025To Backtrack or Not to Backtrack: When Sequential Search Limits Model Reasoningcited by 24
  8. 2025Diffusion LLM with Native Variable Generation Lengths: Let [EOS] Lead the Waycited by 22
  9. 2023Directed Acyclic Transformer Pre-training for High-quality Non-autoregressive Text Generationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026LogicDiff: Logic-Guided Denoising Improves Zero-Shot Reasoning in Masked Diffusion Language Modelscited by 1
  2. 2026Reliable Parallel Decoding in Masked Diffusion Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Masks: Efficient, Flexible Diffusion Language Models via Deletion-Insertion Processes
  2. 2026Dependency-Guided Parallel Decoding in Discrete Diffusion Language Models
  3. 2026Parallelism and Generation Order in Masked Diffusion Language Models: Limits Today, Potential Tomorrow
  4. 2026Representation-based Masked Diffusion Model
  5. 2026Infinite Mask Diffusion for Few-Step Distillation
  6. 2026DAWN: Dependency-Aware Fast Inference for Diffusion LLMs
  7. 2026CoDR: Training-Free Confidence-Drift Remasking for Diffusion Language Models
  8. 2026Rethinking the Generation Order of Block Diffusion Language Models