mlsanityData to 14 Oct 2026

Unlocking Prompt Infilling Capability for Diffusion Language Models

arXiv 2604.03677 ↗First posted 4 Apr 2026cs.CL cs.AIcited by 3

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Masked diffusion language models (dLMs) generate text through bidirectional denoising, yet this capability remains locked for infilling prompts. This limitation is an artifact of the current supervised finetuning (SFT) convention of applying response-only masking. To unlock this capability, we extend full-sequence masking during SFT, where both prompts and responses are masked jointly. Once unlocked, the model infills masked portions of a prompt template conditioned on few-shot examples. We show that such model-infilled prompts match or surpass manually designed templates, transfer effectively across models, and are complementary to existing prompt optimization methods. Our results suggest that training practices, not architectural limitations, are the primary bottleneck preventing masked diffusion language models from infilling effective prompts

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 25 of its 26 references that are in the map. First posted on arXiv 4 Apr 2026.

Builds on

25 of 26 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Dream 7B: Diffusion Large Language Models21 Aug
  2. 2025Flexible-length Text Infilling for Discrete Diffusion Models16 Jun
  3. 2025Large Language Diffusion Models14 Feb
  4. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  5. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution25 Oct

Other references · 20

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  9. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  10. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  11. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  12. 2023Large Language Models as Optimizerscited by 864
8 more
  1. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 832
  2. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
  3. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  4. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
  5. 2024Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programscited by 291
  6. 2020HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verificationcited by 201
  7. 2020Enabling Language Models to Fill in the Blankscited by 122
  8. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 93

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Extracting Training Data from Diffusion Language Models via Infillingcited by 0
  2. 2026Reducing Pretraining-Generation Mismatch in Diffusion Language Modelscited by 0
  3. 2026Noise Your Prompt: Noising Conditioning Tokens in Continuous Diffusion Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Soft-Masked Diffusion Language Models
  2. 2025Activation Steering for Masked Diffusion Language Models
  3. 2026Remask, Don't Replace: Token-to-Mask Refinement in Diffusion Language Models
  4. 2026Not All Denoising Steps Are Equal: Model Scheduling for Faster Masked Diffusion Language Models
  5. 2026PreDiff-LM: Pretrained Discrete Masked Diffusion Language Modeling with Hybrid Attention
  6. 2026Revise, Don't Freeze: Sampler-Matched Training for Self-Correcting Masked Diffusion Language Models
  7. 2026In-Place Instruction Following in Diffusion Language Models
  8. 2026Machine Unlearning for Masked Diffusion Language Models