mlsanityData to 7 Oct 2026

PreDiff-LM: Pretrained Discrete Masked Diffusion Language Modeling with Hybrid Attention

arXiv 2607.25157 ↗First posted 28 Jul 2026cs.AIcited by 0

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Discrete masked diffusion language models support bidirectional generation and infilling, but adapting pretrained autoregressive (AR) transformers requires reconciling causal pretraining with bidirectional denoising. We study this problem at the level of attention rather than claiming AR-weight reuse itself as novel. PreDiff-LM preserves causal attention within the observed prompt while allowing full bidirectional attention within the masked target. Under a matched GPT-2 Medium, WikiText-103, 90K-step setup, this hybrid mask improves unconditional perplexity from 34.1 to 28.7 and MAUVE from 0.71 to 0.78 over uniform bidirectional attention with the same AR initialization. Attention adaptation also composes with a DiffuGPT-style objective adaptation, reaching 26.9 perplexity. Pretrained initialization reduces the steps required to reach perplexity below 50 from about 350K to 8K, although a compute-matched fine-tuned AR model remains stronger at equal scale (18.9 versus 28.7). Beyond perplexity, PreDiff-LM improves repetition, distributional quality, four zero-shot downstream tasks, and human preference over prior diffusion baselines. The results position hybrid attention as a complementary mechanism for adapting pretrained causal backbones, while making explicit the remaining quality and inference-efficiency gaps to optimized AR models.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 23 of its 25 references that are in the map. First posted on arXiv 28 Jul 2026.

Builds on

23 of 25 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuning18 Sep
  2. 2025Dream 7B: Diffusion Large Language Models21 Aug
  3. 2025Large Language Diffusion Models14 Feb
  4. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Models23 Oct
  5. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  6. 2024Simplified and Generalized Masked Diffusion for Discrete Data6 Jun
  7. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data6 Jun
  8. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution25 Oct
  9. 2022Continuous diffusion for categorical data28 Nov
  10. 2021Autoregressive Diffusion Models5 Oct

Other references · 13

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  7. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  8. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  9. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  10. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  11. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  12. 2020Enabling Language Models to Fill in the Blankscited by 121
1 more
  1. 2026Discrete Stochastic Localization for Non-autoregressive Generationcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Retrofitting Linear Attention into Diffusion Language Models
  2. 2026Reducing Pretraining-Generation Mismatch in Diffusion Language Models
  3. 2026dQwen3.5: Hybrid-Attention Diffusion Language Models
  4. 2025Attention Sinks in Diffusion Language Models
  5. 2026Unlocking Prompt Infilling Capability for Diffusion Language Models
  6. 2025Why mask diffusion does not work
  7. 2025Activation Steering for Masked Diffusion Language Models
  8. 2026UNIFUSION: Adapting Autoregressive Language Models into Discrete Diffusion under a Unified Reverse-Rate Objective