# Encoder-Decoder Diffusion Language Models for Efficient Training and Inference

arXiv: https://arxiv.org/abs/2510.22852 · HTML: https://mlsanity.com/paper/2510.22852/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in Discrete and text diffusion, one of 14 directions in Diffusion models.
- Builds on 59 of its 79 references that are in the map.
- First posted on arXiv 26 Oct 2025.

## Placed in

- Diffusion models / [Discrete and text diffusion](https://mlsanity.com/diffusion/discrete-and-text-diffusion/)

## Builds on (most cited first, 30 of 59)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Structured Denoising Diffusion Models in Discrete State-Spaces](https://mlsanity.com/paper/2107.03006/) (2021)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [Diffusion-LM Improves Controllable Text Generation](https://mlsanity.com/paper/2205.14217/) (2022)
- [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023)
- [The LAMBADA dataset: Word prediction requiring a broad discourse context](https://mlsanity.com/paper/1606.06031/) (2016)
- [Large Language Diffusion Models](https://mlsanity.com/paper/2502.09992/) (2025)
- [Simple and Effective Masked Diffusion Language Models](https://mlsanity.com/paper/2406.07524/) (2024)
- [Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads](https://mlsanity.com/paper/2401.10774/) (2024)
- [Language Modeling with Gated Convolutional Networks](https://mlsanity.com/paper/1612.08083/) (2016)
- [Scaling Vision Transformers to 22 Billion Parameters](https://mlsanity.com/paper/2302.05442/) (2023)

## Built on by (most cited first, 24 of 24)

- [ELF: Embedded Language Flows](https://mlsanity.com/paper/2605.10938/) (2026)
- [Learning Unmasking Policies for Diffusion Language Models](https://mlsanity.com/paper/2512.09106/) (2025)
- [Stop Training for the Worst: Progressive Unmasking Accelerates Masked Diffusion Training](https://mlsanity.com/paper/2602.10314/) (2026)
- [DUEL: Exact Likelihood for Masked Diffusion via Deterministic Unmasking](https://mlsanity.com/paper/2603.01367/) (2026)
- [Causality in Video Diffusers is Separable from Denoising](https://mlsanity.com/paper/2602.10095/) (2026)
- [Nemotron-Labs-TwoTower: Diffusion Language Modeling with Pretrained Autoregressive Context](https://mlsanity.com/paper/2606.26493/) (2026)
- [FlashBlock: Attention Caching for Efficient Long-Context Block Diffusion](https://mlsanity.com/paper/2602.05305/) (2026)
- [Locally Coherent Parallel Decoding in Diffusion Language Models](https://mlsanity.com/paper/2603.20216/) (2026)
- [DeltaFlow: Noise-Adaptive Bidirectional Gated Delta Networks for Embedded Language Flows](https://mlsanity.com/paper/2608.01240/) (2026)
- [BARD: Bridging AutoRegressive and Diffusion Vision-Language Models Via Highly Efficient Progressive Block Merging and Stage-Wise Distillation](https://mlsanity.com/paper/2604.16514/) (2026)
- [Diffusion Language Models: An Experimental Analysis](https://mlsanity.com/paper/2606.19475/) (2026)
- [CRoCoDiL: Continuous and Robust Conditioned Diffusion for Language](https://mlsanity.com/paper/2603.20210/) (2026)
- [Not All Denoising Steps Are Equal: Model Scheduling for Faster Masked Diffusion Language Models](https://mlsanity.com/paper/2604.02340/) (2026)
- [Accelerating Masked Diffusion Large Language Models: A Survey of Efficient Inference Techniques](https://mlsanity.com/paper/2607.12829/) (2026)
- [Sentence Curve Language Models](https://mlsanity.com/paper/2602.01807/) (2026)
- [Latent Block-Diffusion Temporal Point Processes: A Semi-Autoregressive Framework for Asynchronous Event Sequence Generation](https://mlsanity.com/paper/2606.24982/) (2026)
- [Neuromorphic Diffusion Language Models: Addressing Compute and Memory Bottlenecks via Sparsity and Block Denoising](https://mlsanity.com/paper/2607.24841/) (2026)
- [SEED: Self-Speculative Decoding via Implicit Encoder-Decoder](https://mlsanity.com/paper/2609.36590/) (2026)
- [Gumbel Straight Flow: Distilling Autoregressive Models into One-step Flow Maps](https://mlsanity.com/paper/2610.00497/) (2026)
- [Clock Diffusion: Efficient Semi-Autoregressive Continuous Diffusion Language Models](https://mlsanity.com/paper/2610.00894/) (2026)
- [Language Models as AI Research World Models](https://mlsanity.com/paper/2610.12235/) (2026)
- [One-Step Text Generation by Seq-Drifting](https://mlsanity.com/paper/2610.12879/) (2026)
- [Next-Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generation](https://mlsanity.com/paper/2610.14162/) (2026)
- [Revisiting the 16x Pretraining Compute Gap between Diffusion and Autoregressive Language Models](https://mlsanity.com/paper/2610.17590/) (2026)

## Abstract (arXiv)

Discrete diffusion models enable parallel token sampling for faster inference than autoregressive approaches. However, prior diffusion models use a decoder-only architecture, which requires sampling algorithms that invoke the full network at every denoising step and incur high computational cost. Our key insight is that discrete diffusion models perform two types of computation: 1) representing clean tokens and 2) denoising corrupted tokens, which enables us to use separate modules for each task. We propose an encoder-decoder architecture to accelerate discrete diffusion inference, which relies on an encoder to represent clean tokens and a lightweight decoder to iteratively refine a noised sequence. We also show that this architecture enables faster training of block diffusion models, which partition sequences into blocks for better quality and are commonly used in diffusion language model inference. We introduce a framework for Efficient Encoder-Decoder Diffusion (E2D2), consisting of an architecture with specialized training and sampling algorithms, and we show that E2D2 achieves superior trade-offs between generation quality and inference throughput on summarization, translation, and mathematical reasoning tasks. We provide the code, model weights, and blog post on the project page: https://m-arriola.com/e2d2