mlsanityData to 14 Oct 2026

Anchored Diffusion Language Model

arXiv 2505.18456 ↗First posted 24 May 2025cs.CL cs.LGcited by 17

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Diffusion Language Models (DLMs) promise parallel generation and bidirectional context, yet they underperform autoregressive (AR) models in both likelihood modeling and generated text quality. We identify that this performance gap arises when important tokens (e.g., key words or low-frequency words that anchor a sentence) are masked early in the forward process, limiting contextual information for accurate reconstruction. To address this, we introduce the Anchored Diffusion Language Model (ADLM), a novel two-stage framework that first predicts distributions over important tokens via an anchor network, and then predicts the likelihoods of missing tokens conditioned on the anchored predictions. ADLM significantly improves test perplexity on LM1B and OpenWebText, achieving up to 25.4% gains over prior DLMs, and narrows the gap with strong AR baselines. It also achieves state-of-the-art performance in zero-shot generalization across seven benchmarks and surpasses AR models in MAUVE score, which marks the first time a DLM generates better human-like text than an AR model. Theoretically, we derive an Anchored Negative Evidence Lower Bound (ANELBO) objective and show that anchoring improves sample complexity and likelihood modeling. Beyond diffusion, anchoring boosts performance in AR models and enhances reasoning in math and logic tasks, outperforming existing chain-of-thought approaches

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 47 of its 59 references that are in the map. First posted on arXiv 24 May 2025.

Builds on

47 of 59 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models12 Mar
  2. 2025Generalized Interpolating Discrete Diffusion6 Mar
  3. 2025Remasking Discrete Diffusion Models with Inference-Time Scaling1 Mar
  4. 2025Large Language Diffusion Models14 Feb
  5. 2024Scaling up Masked Diffusion Models on Text24 Oct
  6. 2024Think While You Generate: Discrete Diffusion with Planned Denoising8 Oct
  7. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  8. 2024Simplified and Generalized Masked Diffusion for Discrete Data6 Jun
  9. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data6 Jun
  10. 2024Glauber Generative Model: Discrete Diffusion Models via Binary Classification27 May
  11. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution25 Oct
  12. 2023Likelihood-Based Diffusion Language Models30 May
1 more
  1. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models28 Nov

Other references · 34

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  7. 2022Scalable Diffusion Models with Transformerscited by 6,054
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  9. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  10. 2016Pointer Sentinel Mixture Modelscited by 3,096
  11. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  12. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
22 more
  1. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  2. 2018Hierarchical Neural Story Generationcited by 1,193
  3. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  4. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,027
  5. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  6. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  7. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
  8. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 797
  9. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 478
  10. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 470
  11. 2022A Continuous Time Framework for Discrete Denoising Modelscited by 451
  12. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 354
  13. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thoughtcited by 337
  14. 2024Discrete Flow Matchingcited by 317
  15. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 308
  16. 2023Think before you speak: Training Language Models With Pause Tokenscited by 264
  17. 2024From Explicit CoT to Implicit CoT: Learning to Internalize CoT Step by Stepcited by 235
  18. 2025CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillationcited by 225
  19. 2015Visualizing and Understanding Neural Models in NLPcited by 200
  20. 2019BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Modelcited by 185
  21. 2025Reasoning to Learn from Latent Thoughtscited by 45
  22. 2021OmniNet: Omnidirectional Representations from Transformerscited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 11.

17 papers.

Sort:

By area:

  1. 2025A Survey on Diffusion Language Modelscited by 78
  2. 2025Any-Order Flexible Length Masked Diffusioncited by 51
  3. 2025Fine-Tuning Masked Diffusion for Provable Self-Correctioncited by 39
  4. 2025Enabling Approximate Joint Sampling in Diffusion LMscited by 10
  5. 2026Context Tokens are Anchors: Understanding the Repetition Curse in dMLLMs from an Information Flow Perspectivecited by 6
  6. 2025Test-Time Anchoring for Discrete Diffusion Posterior Samplingcited by 5
  7. 2026TABES: Trajectory-Aware Backward-on-Entropy Steering for Masked Diffusion Modelscited by 3
  8. 2026Discrete Stochastic Localization for Non-autoregressive Generationcited by 3
  9. 2026AnCoder: Anchored Code Generation via Discrete Diffusion Modelscited by 1
  10. 2026Dystruct: Dynamically Structured Diffusion Language Model Decoding via Bayesian Inferencecited by 1
  11. 2026DSL-LLaDA: Scaling Continuous Denoising to 8B Masked Diffusion LMscited by 1
  12. 2025Syntax-Guided Diffusion Language Models with User-Integrated Personalizationcited by 0
5 more
  1. 2026LaDA-Band: Language Diffusion Models for Vocal-to-Accompaniment Generationcited by 0
  2. 2026Discrete Stochastic Localization for Non-autoregressive Generationcited by 0
  3. 2026Discretizing Continuous Time Series for Imputation with Masked Diffusion Trainingcited by 0
  4. 2026dQwen3.5: Hybrid-Attention Diffusion Language Modelscited by 0
  5. 2026Time-Anchored Diffusion Language Models: Latent-Space Caching for Fast Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Models
  2. 2026Diffusion Language Models: An Experimental Analysis
  3. 2026Revisiting the 16x Pretraining Compute Gap between Diffusion and Autoregressive Language Models
  4. 2026Top 10 Open Challenges Steering the Future of Diffusion Language Model and Its Variants
  5. 2025Breaking AR's Sampling Bottleneck: Provable Acceleration via Diffusion Language Models
  6. 2026Accelerating Masked Diffusion Large Language Models: A Survey of Efficient Inference Techniques
  7. 2026From AR to Diffusion: Efficiently Adapting Large Language Models with Strictly Causal and Elastic Horizons
  8. 2026Differences in Text Generated by Diffusion and Autoregressive Language Models