mlsanityData to 7 Oct 2026

A Cheaper and Better Diffusion Language Model with Soft-Masked Noise

arXiv 2304.04746 ↗First posted 10 Apr 2023cs.CL cs.AIcited by 21

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Diffusion models that are based on iterative denoising have been recently proposed and leveraged in various generation tasks like image generation. Whereas, as a way inherently built for continuous data, existing diffusion models still have some limitations in modeling discrete data, e.g., languages. For example, the generally used Gaussian noise can not handle the discrete corruption well, and the objectives in continuous spaces fail to be stable for textual data in the diffusion process especially when the dimension is high. To alleviate these issues, we introduce a novel diffusion model for language modeling, Masked-Diffuse LM, with lower training cost and better performances, inspired by linguistic features in languages. Specifically, we design a linguistic-informed forward process which adds corruptions to the text through strategically soft-masking to better noise the textual data. Also, we directly predict the categorical distribution with cross-entropy loss function in every diffusion step to connect the continuous space and discrete space in a more efficient and straightforward way. Through experiments on 5 controlled generation tasks, we demonstrate that our Masked-Diffuse LM can achieve better generation quality than the state-of-the-art diffusion models with better efficiency.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 48 of its 59 references that are in the map. First posted on arXiv 10 Apr 2023.

Builds on

48 of 59 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023DINOISER: Diffused Conditional Sequence Learning by Manipulating Noises20 Feb
  2. 2023A Reparameterized Discrete Diffusion Model for Text Generation11 Feb
  3. 2022Continuous diffusion for categorical data28 Nov
  4. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models28 Nov
  5. 2022DiffusER: Discrete Diffusion via Edit-based Reconstruction30 Oct
  6. 2022DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models17 Oct
  7. 2021Autoregressive Diffusion Models5 Oct

Other references · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  9. 2020Denoising Diffusion Implicit Modelscited by 9,655
  10. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  11. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  12. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
29 more
  1. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  2. 2022Generative Adversarial Networkscited by 5,603
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2013Auto-Encoding Variational Bayescited by 3,980
  5. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  6. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  7. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  8. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  9. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  10. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  11. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  12. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  13. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  14. 2021Argmax Flows and Multinomial Diffusion: Learning Categorical Distributionscited by 566
  15. 2017Non-Autoregressive Neural Machine Translationcited by 457
  16. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 434
  17. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 377
  18. 2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 374
  19. 2019ERNIE: Enhanced Representation through Knowledge Integrationcited by 301
  20. 2017The E2E Dataset: New Challenges For End-to-End Generationcited by 252
  21. 2020UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Trainingcited by 204
  22. 2021Step-unrolled Denoising Autoencoders for Text Generationcited by 102
  23. 2020Non-Autoregressive Machine Translation with Latent Alignmentscited by 96
  24. 2020Multi-View Sequence-to-Sequence Models with Conversational Structure for Abstractive Dialogue Summarizationcited by 72
  25. 2021Sampling-Based Approximations to Minimum Bayes Risk Decoding for Neural Machine Translationcited by 63
  26. 2021Structure-Aware Abstractive Conversation Summarization via Discourse and Action Graphscited by 36
  27. 2020A Study of Non-autoregressive Model for Sequence Generationcited by 31
  28. 2021TF-IDF vs Word Embeddings for Morbidity Identification in Clinical Notes: An Initial Studycited by 7
  29. 2016The word entropy of natural languagescited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 5; 2025: 6; 2026: 5.

21 papers.

Sort:

By area:

  1. 2024Diffusion Language Models Are Versatile Protein Learnerscited by 93
  2. 2025Discrete Diffusion in Large Language and Multimodal Models: A Surveycited by 61
  3. 2023Diffusion Language Models Can Perform Many Tasks with Scaling and Instruction-Finetuningcited by 39
  4. 2023DiffuSeq-v2: Bridging Discrete and Continuous Text Spaces for Accelerated Seq2Seq Diffusion Modelscited by 36
  5. 2025Efficient Diffusion Models: A Surveycited by 35
  6. 2023A Survey of Diffusion Models in Natural Language Processingcited by 16
  7. 2024DiffLM: Controllable Synthetic Data Generation via Diffusion Language Modelscited by 9
  8. 2025Blockwise SFT for Diffusion Language Models: Reconciling Bidirectional Attention and Autoregressive Decodingcited by 9
  9. 2023David helps Goliath: Inference-Time Collaboration Between Small Specialized and Large General Diffusion LMscited by 6
  10. 2023InfoDiffusion: Information Entropy Aware Diffusion Process for Non-Autoregressive Text Generationcited by 5
  11. 2024Conditional [MASK] Discrete Diffusion Language Modelcited by 4
  12. 2025Unifying Continuous and Discrete Text Diffusion with Non-simultaneous Diffusion Processescited by 4
9 more
  1. 2025DiffuMamba: High-Throughput Diffusion LMs with Mamba Backbonecited by 4
  2. 20243D Object Detection from Point Cloud via Voting Step Diffusioncited by 3
  3. 2026Breaking Block Boundaries: Anchor-based History-stable Decoding for Diffusion Large Language Modelscited by 3
  4. 2026FastDiSS: Few-step Match Many-step Diffusion Language Model on Sequence-to-Sequence Generation--Full Versioncited by 2
  5. 2026Towards Latent Diffusion Suitable For Textcited by 1
  6. 2026DSL-LLaDA: Scaling Continuous Denoising to 8B Masked Diffusion LMscited by 1
  7. 2026EchoChange: A Diffusion Language Model with Dual Pass Remasking for Factual Remote Sensing Disaster Change Captioningcited by 1
  8. 2024Discrete Modeling via Boundary Conditional Diffusion Processescited by 0
  9. 2025SA-DiffuSeq: Addressing Computational and Scalability Challenges in Long-Document Generation with Sparse Attentioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Simple and Effective Masked Diffusion Language Models
  2. 2024Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling
  3. 2026Not All Denoising Steps Are Equal: Model Scheduling for Faster Masked Diffusion Language Models
  4. 2025Soft-Masked Diffusion Language Models
  5. 2025Simple Denoising Diffusion Language Models
  6. 2025Discrete Diffusion Models for Language Generation
  7. 2025Improving Text Style Transfer using Masked Diffusion Language Models with Inference-time Scaling
  8. 2026DLM-SWAI: Steering Diffusion Language Models Before They Unmask