mlsanityData to 7 Oct 2026

Extracting Training Data from Diffusion Language Models via Infilling

arXiv 2605.24173 ↗First posted 22 May 2026cs.CL cs.AI cs.CRcited by 0

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Memorization in large language models has been studied almost exclusively through prefix-conditioned extraction, a natural choice for autoregressive models. However, diffusion language models (DLMs) can denoise masked tokens at arbitrary positions. Thus, prefix-only probing reveals only one facet of memorization in DLMs and significantly underestimates the risk of training-data extraction. In order to realistically model extractability of training data in DLMs, we introduce infilling extraction, a data-extraction protocol parameterized by an arbitrary binary mask that subsumes prefix-only probing and accounts for the bidirectional inductive bias of DLMs. Instantiating it on LLaDA-8B and Dream-7B across five extraction modes, three training pipelines, and three corpora covering verbatim and partial leakage, we find that mask geometry governs extractability: edge-conditioned masks extract up to three times more verbatim sequences than prefix-conditioned ones, and bidirectional access opens channels inaccessible in autoregressive models. In particular, we show that a realistic adversary with access to training data where personally identifiable information has been redacted, can even achieve higher recall on extracting redacted email addresses from DLMs than from scale-matched autoregressive models. Tunable parameters for decoding measurably affect extraction performance, while a follow-up supervised finetuning stage does not eliminate the prior memorization.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 42 of its 50 references that are in the map. First posted on arXiv 22 May 2026.

Builds on

42 of 50 references.

Closest ancestors · 14

Same direction, newest first

  1. 2026Unlocking Prompt Infilling Capability for Diffusion Language Models4 Apr
  2. 2026dLLM: Simple Diffusion Language Modeling26 Feb
  3. 2026LLaDA2.1: Speeding Up Text Diffusion via Token Editing9 Feb
  4. 2025LLaDA2.0: Scaling Up Diffusion Language Models to 100B10 Dec
  5. 2025Dream 7B: Diffusion Large Language Models21 Aug
  6. 2025A Survey on Diffusion Language Models14 Aug
  7. 2025Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference4 Aug
  8. 2025Breaking AR's Sampling Bottleneck: Provable Acceleration via Diffusion Language Models27 May
  9. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models12 Mar
  10. 2025Large Language Diffusion Models14 Feb
  11. 2024Energy-Based Diffusion Language Models for Text Generation28 Oct
  12. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Models23 Oct
2 more
  1. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models28 Nov
  2. 2022Self-conditioned Embedding Diffusion for Text Generation8 Nov

Other references · 28

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2020Extracting Training Data from Large Language Modelscited by 2,144
  4. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  5. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  6. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  7. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  8. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  9. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 404
  10. 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 278
  11. 2025Mercury: Ultra-Fast Language Models Based on Diffusioncited by 225
  12. 2024Do Membership Inference Attacks Work on Large Language Models?cited by 188
16 more
  1. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learningcited by 178
  2. 2023Copyright Violations and Large Language Modelscited by 108
  3. 2025Accelerating Diffusion LLMs via Adaptive Parallel Decodingcited by 108
  4. 2023Bag of Tricks for Training Data Extraction from Language Modelscited by 67
  5. 2023Exploring Memorization in Fine-tuned Language Modelscited by 44
  6. 2024Measuring memorization in language models via probabilistic extractioncited by 40
  7. 2025The Devil behind the mask: An emergent safety vulnerability of Diffusion LLMscited by 32
  8. 2025Jailbreaking Large Language Diffusion Models: Revealing Hidden Safety Flaws in Diffusion-Based Text Generationcited by 24
  9. 2026DynHD: Hallucination Detection for Diffusion Large Language Models via Denoising Dynamics Deviation Learningcited by 13
  10. 2025The Landscape of Memorization in LLMs: Mechanisms, Measurement, and Mitigationcited by 9
  11. 2025TraceDet: Hallucination Detection from the Decoding Trace of Diffusion Large Language Modelscited by 7
  12. 2025Partition Generative Modeling: Masked Modeling Without Maskscited by 6
  13. 2026Membership Inference Attacks Against Fine-tuned Diffusion Language Modelscited by 5
  14. 2026Characterizing Memorization in Diffusion Language Models: Generalized Extraction and Sampling Effectscited by 3
  15. 2025Balancing Innovation and Oversight: AI in the U.S. Treasury and IRS: A Surveycited by 1
  16. 2025Exploring Cross-Client Memorization of Training Data in Large Language Models for Federated Learningcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Simple and Effective Masked Diffusion Language Models
  2. 2026Reducing Pretraining-Generation Mismatch in Diffusion Language Models
  3. 2026Understanding and Accelerating the Training of Masked Diffusion Language Models
  4. 2026DreamingGoose: Staged Distillation from Autoregressive Transformers to Bidirectional Recurrent Diffusion Language Models
  5. 2024Scaling up Masked Diffusion Models on Text
  6. 2026CoDR: Training-Free Confidence-Drift Remasking for Diffusion Language Models
  7. 2026DiLaDiff: Distilled Latent-Augmented Diffusion for Language Modeling
  8. 2025Soft-Masked Diffusion Language Models