# Diffusion Guided Language Modeling

arXiv: https://arxiv.org/abs/2408.04220 · HTML: https://mlsanity.com/paper/2408.04220/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Discrete and text diffusion, one of 14 directions in Diffusion models.
- Builds on 47 of its 58 references that are in the map.
- First posted on arXiv 8 Aug 2024.

## Placed in

- Diffusion models / [Discrete and text diffusion](https://mlsanity.com/diffusion/discrete-and-text-diffusion/)

## Builds on (most cited first, 30 of 47)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [Diffusion Posterior Sampling for General Noisy Inverse Problems](https://mlsanity.com/paper/2209.14687/) (2022)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)
- [Variational Diffusion Models](https://mlsanity.com/paper/2107.00630/) (2021)
- [Diffusion-LM Improves Controllable Text Generation](https://mlsanity.com/paper/2205.14217/) (2022)
- [eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers](https://mlsanity.com/paper/2211.01324/) (2022)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [Scaling Vision Transformers to 22 Billion Parameters](https://mlsanity.com/paper/2302.05442/) (2023)
- [MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers](https://mlsanity.com/paper/2102.01454/) (2021)
- [Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models](https://mlsanity.com/paper/2108.08877/) (2021)

## Built on by (most cited first, 15 of 15)

- [Scaling up Masked Diffusion Models on Text](https://mlsanity.com/paper/2410.18514/) (2024)
- [Large Concept Models: Language Modeling in a Sentence Representation Space](https://mlsanity.com/paper/2412.08821/) (2024)
- [Theoretical Benefit and Limitation of Diffusion Language Model](https://mlsanity.com/paper/2502.09622/) (2025)
- [ELF: Embedded Language Flows](https://mlsanity.com/paper/2605.10938/) (2026)
- [LaDiR: Latent Diffusion Enhances LLMs for Text Reasoning](https://mlsanity.com/paper/2510.04573/) (2025)
- [Adaptive Classifier-Free Guidance via Dynamic Low-Confidence Masking](https://mlsanity.com/paper/2505.20199/) (2025)
- [Learning a Generative Meta-Model of LLM Activations](https://mlsanity.com/paper/2602.06964/) (2026)
- [Language Models Are Implicitly Continuous](https://mlsanity.com/paper/2504.03933/) (2025)
- [From Text to Talk: Audio-Language Model Needs Non-Autoregressive Joint Training](https://mlsanity.com/paper/2509.20072/) (2025)
- [Simple Denoising Diffusion Language Models](https://mlsanity.com/paper/2510.22926/) (2025)
- [Stop-Think-AutoRegress: Language Modeling with Latent Diffusion Planning](https://mlsanity.com/paper/2602.20528/) (2026)
- [HIVE: Hidden-Evidence Verification for Hallucination Detection in Diffusion Large Language Models](https://mlsanity.com/paper/2604.26139/) (2026)
- [Next-Token Prediction Should be Ambiguity-Sensitive: A Meta-Learning Perspective](https://mlsanity.com/paper/2506.16288/) (2025)
- [Focus on the Core: Empowering Diffusion Large Language Models by Self-Contrast](https://mlsanity.com/paper/2605.01373/) (2026)
- [Repurposing Pre-trained LLMs as High Fidelity Continuous Text Autoencoders](https://mlsanity.com/paper/2609.27248/) (2026)

## Abstract (arXiv)

Current language models demonstrate remarkable proficiency in text generation. However, for many applications it is desirable to control attributes, such as sentiment, or toxicity, of the generated language -- ideally tailored towards each specific use case and target audience. For auto-regressive language models, existing guidance methods are prone to decoding errors that cascade during generation and degrade performance. In contrast, text diffusion models can easily be guided with, for example, a simple linear sentiment classifier -- however they do suffer from significantly higher perplexity than auto-regressive alternatives. In this paper we use a guided diffusion model to produce a latent proposal that steers an auto-regressive language model to generate text with desired properties. Our model inherits the unmatched fluency of the auto-regressive approach and the plug-and-play flexibility of diffusion. We show that it outperforms previous plug-and-play guidance methods across a wide range of benchmark data sets. Further, controlling a new attribute in our framework is reduced to training a single logistic regression classifier.