mlsanityData to 7 Oct 2026

Representation-based Masked Diffusion Model

arXiv 2609.12382 ↗First posted 11 Sep 2026cs.CL stat.MLcited by 0

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Masked Diffusion Models (MDMs) have emerged as a compelling paradigm for language modeling, offering the capability for efficient parallel text generation. However, existing parallel sampling methods typically update multiple masked tokens independently and ignore the complex mutual dependencies among the masked tokens. This independent updating mechanism lacks global coordination and might lead to incoherent outputs. To address this limitation, we propose Representation-based Masked Diffusion Model (RMDM), a framework that leverages the text representation to explicitly encode global semantics and help to parallel update tokens more precisely. Specifically, we first encode text into a continuous semantic space using a pretrained encoder and learn an invertible transformation that normalizes the representation distribution to a Gaussian prior, facilitating efficient sampling during generation. Conditioned on this latent semantic representation, we train a masked diffusion model to learn the conditional text distribution, where the representation serves as global semantic guidance to coordinate parallel token updates and faithfully approximate the target distribution. Empirical results demonstrate that RMDM significantly improves generation quality, particularly in aggressive few-step sampling regimes.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 37 of its 66 references that are in the map. First posted on arXiv 11 Sep 2026.

Builds on

37 of 66 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Coevolutionary Continuous Discrete Diffusion: Make Your Diffusion Language Model a Latent Reasoner3 Oct
  2. 2025The Diffusion Duality12 Jun
  3. 2025Generalized Interpolating Discrete Diffusion6 Mar
  4. 2024Simplified and Generalized Masked Diffusion for Discrete Data6 Jun
  5. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution25 Oct
  6. 2023TESS: Text-to-Text Self-Conditioned Simplex Diffusion15 May
  7. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models28 Nov
  8. 2022SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Control31 Oct
  9. 2022DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models17 Oct

Other references · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  6. 2016Pointer Sentinel Mixture Modelscited by 2,992
  7. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  8. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  9. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  10. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
  11. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  12. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
16 more
  1. 2015Generating Sentences from a Continuous Spacecited by 607
  2. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
  3. 2025Mean Flows for One-step Generative Modelingcited by 524
  4. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 477
  5. 2017Non-Autoregressive Neural Machine Translationcited by 457
  6. 2022A Continuous Time Framework for Discrete Denoising Modelscited by 437
  7. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 404
  8. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  9. 2024Generative Flows on Discrete State-Spaces: Enabling Multimodal Flows with Applications to Protein Co-Designcited by 294
  10. 2019Understanding and Improving Layer Normalizationcited by 213
  11. 2022Concrete Score Matching: Generalized Score Matching for Discrete Datacited by 144
  12. 2018Fast Decoding in Sequence Models using Discrete Latent Variablescited by 103
  13. 2024Flow Matching with General Discrete Paths: A Kinetic-Optimal Perspectivecited by 54
  14. 2024Learning Iterative Reasoning through Energy Diffusioncited by 36
  15. 2025Latent Thought Models with Variational Bayes Inference-Time Computationcited by 25
  16. 2025Diffuse Everything: Multimodal Diffusion Models on Arbitrary State Spacescited by 19

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Scaling up Masked Diffusion Models on Text
  2. 2024Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling
  3. 2026Infinite Mask Diffusion for Few-Step Distillation
  4. 2026Auto-Regressive Masked Diffusion Models
  5. 2025Parallel Sampling from Masked Diffusion Models via Conditional Independence Testing
  6. 2026CRoCoDiL: Continuous and Robust Conditioned Diffusion for Language
  7. 2025Theoretical Benefit and Limitation of Diffusion Language Model
  8. 2026Beyond Masks: Efficient, Flexible Diffusion Language Models via Deletion-Insertion Processes