# Unifying Autoregressive and Diffusion-Based Sequence Generation

arXiv: https://arxiv.org/abs/2504.06416 · HTML: https://mlsanity.com/paper/2504.06416/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 40 references that are in the map.
- First posted on arXiv 8 Apr 2025.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Structured Denoising Diffusion Models in Discrete State-Spaces](https://mlsanity.com/paper/2107.03006/) (2021)
- [Diffusion Models: A Comprehensive Survey of Methods and Applications](https://mlsanity.com/paper/2209.00796/) (2022)
- [Large Language Diffusion Models](https://mlsanity.com/paper/2502.09992/) (2025)
- [Simple and Effective Masked Diffusion Language Models](https://mlsanity.com/paper/2406.07524/) (2024)
- [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://mlsanity.com/paper/1909.05858/) (2019)
- [Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution](https://mlsanity.com/paper/2310.16834/) (2023)
- [Simplified and Generalized Masked Diffusion for Discrete Data](https://mlsanity.com/paper/2406.04329/) (2024)
- [MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers](https://mlsanity.com/paper/2102.01454/) (2021)
- [Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models](https://mlsanity.com/paper/2503.09573/) (2025)
- [Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data](https://mlsanity.com/paper/2406.03736/) (2024)
- [One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling](https://mlsanity.com/paper/1312.3005/) (2013)
- [Cold Diffusion: Inverting Arbitrary Image Transforms Without Noise](https://mlsanity.com/paper/2208.09392/) (2022)
- [DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models](https://mlsanity.com/paper/2211.15029/) (2022)
- [Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions](https://mlsanity.com/paper/2502.06768/) (2025)
- [Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling](https://mlsanity.com/paper/2409.02908/) (2024)
- [Remasking Discrete Diffusion Models with Inference-Time Scaling](https://mlsanity.com/paper/2503.00307/) (2025)
- [Likelihood-Based Diffusion Language Models](https://mlsanity.com/paper/2305.18619/) (2023)
- [The Matrix: Infinite-Horizon World Generation with Real-Time Moving Control](https://mlsanity.com/paper/2412.03568/) (2024)
- [Path Planning for Masked Diffusion Model Sampling](https://mlsanity.com/paper/2502.03540/) (2025)
- [Think While You Generate: Discrete Diffusion with Planned Denoising](https://mlsanity.com/paper/2410.06264/) (2024)
- [OmniNet: Omnidirectional Representations from Transformers](https://mlsanity.com/paper/2103.01075/) (2021)
- [Diffusion-NAT: Self-Prompting Discrete Diffusion for Non-Autoregressive Text Generation](https://mlsanity.com/paper/2305.04044/) (2023)

## Built on by (most cited first, 17 of 17)

- [DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation](https://mlsanity.com/paper/2506.20639/) (2025)
- [SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generation](https://mlsanity.com/paper/2510.06303/) (2025)
- [Don't Settle Too Early: Self-Reflective Remasking for Diffusion Language Models](https://mlsanity.com/paper/2509.23653/) (2025)
- [Multiverse: Your Language Models Secretly Decide How to Parallelize and Merge Generation](https://mlsanity.com/paper/2506.09991/) (2025)
- [Set Block Decoding is a Language Model Inference Accelerator](https://mlsanity.com/paper/2509.04185/) (2025)
- [FS-DFM: Fast and Accurate Long Text Generation with Few-Step Diffusion Language Models](https://mlsanity.com/paper/2509.20624/) (2025)
- [Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuning](https://mlsanity.com/paper/2509.15188/) (2025)
- [DiffuMamba: High-Throughput Diffusion LMs with Mamba Backbone](https://mlsanity.com/paper/2511.15927/) (2025)
- [DARE: Diffusion Large Language Models Alignment and Reinforcement Executor](https://mlsanity.com/paper/2604.04215/) (2026)
- [Learning from the Self-future: On-policy Self-distillation for dLLMs](https://mlsanity.com/paper/2606.18195/) (2026)
- [BiTrajDiff: Bidirectional Trajectory Generation with Diffusion Models for Offline Reinforcement Learning](https://mlsanity.com/paper/2506.05762/) (2025)
- [ECHO: Efficient Chest X-ray Report Generation with One-step Block Diffusion](https://mlsanity.com/paper/2604.09450/) (2026)
- [Balancing Understanding and Generation in Discrete Diffusion Models](https://mlsanity.com/paper/2602.01362/) (2026)
- [DICE: Diffusion Large Language Models Excel at Generating CUDA Kernels](https://mlsanity.com/paper/2602.11715/) (2026)
- [Gumbel Distillation for Parallel Text Generation](https://mlsanity.com/paper/2603.22216/) (2026)
- [Foundation Models for Medical Imaging: Status, Challenges, and Directions](https://mlsanity.com/paper/2602.15913/) (2026)
- [Destruction is a General Strategy to Learn Generation; Diffusion's Strength is to Take it Seriously; Exploration is the Future](https://mlsanity.com/paper/2605.30553/) (2026)

## Abstract (arXiv)

We present significant extensions to diffusion-based sequence generation models, blurring the line with autoregressive language models. We introduce hyperschedules, which assign distinct noise schedules to individual token positions, generalizing both autoregressive models (e.g., GPT) and conventional diffusion models (e.g., SEDD, MDLM) as special cases. Second, we propose two hybrid token-wise noising processes that interpolate between absorbing and uniform processes, enabling the model to fix past mistakes, and we introduce a novel inference algorithm that leverages this new feature in a simplified context inspired from MDLM. To support efficient training and inference, we design attention masks compatible with KV-caching. Our methods achieve state-of-the-art perplexity and generate diverse, high-quality sequences across standard benchmarks, suggesting a promising path for autoregressive diffusion-based sequence generation. See code and resources at https://hdlm-colm.github.io/