mlsanityData to 14 Oct 2026

Unifying Autoregressive and Diffusion-Based Sequence Generation

arXiv 2504.06416 ↗First posted 8 Apr 2025cs.LGcited by 17

Not on the map. Why?

Abstract (arXiv)

We present significant extensions to diffusion-based sequence generation models, blurring the line with autoregressive language models. We introduce hyperschedules, which assign distinct noise schedules to individual token positions, generalizing both autoregressive models (e.g., GPT) and conventional diffusion models (e.g., SEDD, MDLM) as special cases. Second, we propose two hybrid token-wise noising processes that interpolate between absorbing and uniform processes, enabling the model to fix past mistakes, and we introduce a novel inference algorithm that leverages this new feature in a simplified context inspired from MDLM. To support efficient training and inference, we design attention masks compatible with KV-caching. Our methods achieve state-of-the-art perplexity and generate diverse, high-quality sequences across standard benchmarks, suggesting a promising path for autoregressive diffusion-based sequence generation. See code and resources at https://hdlm-colm.github.io/

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 27 of its 40 references that are in the map. First posted on arXiv 8 Apr 2025.

Builds on

27 of 40 references.

References · 27

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Scalable Diffusion Models with Transformerscited by 6,054
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  6. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  7. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,114
  8. 2025Large Language Diffusion Modelscited by 912
  9. 2024Simple and Effective Masked Diffusion Language Modelscited by 848
  10. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 762
  11. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 745
  12. 2024Simplified and Generalized Masked Diffusion for Discrete Datacited by 626
15 more
  1. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 470
  2. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Modelscited by 409
  3. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Datacited by 381
  4. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 308
  5. 2022Cold Diffusion: Inverting Arbitrary Image Transforms Without Noisecited by 276
  6. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Modelscited by 214
  7. 2025Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusionscited by 214
  8. 2024Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Samplingcited by 208
  9. 2025Remasking Discrete Diffusion Models with Inference-Time Scalingcited by 189
  10. 2023Likelihood-Based Diffusion Language Modelscited by 151
  11. 2024The Matrix: Infinite-Horizon World Generation with Real-Time Moving Controlcited by 80
  12. 2025Path Planning for Masked Diffusion Model Samplingcited by 80
  13. 2024Think While You Generate: Discrete Diffusion with Planned Denoisingcited by 63
  14. 2021OmniNet: Omnidirectional Representations from Transformerscited by 23
  15. 2023Diffusion-NAT: Self-Prompting Discrete Diffusion for Non-Autoregressive Text Generationcited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 8.

17 papers.

Sort:

By area:

  1. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generationcited by 187
  2. 2025SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generationcited by 129
  3. 2025Don't Settle Too Early: Self-Reflective Remasking for Diffusion Language Modelscited by 37
  4. 2025Multiverse: Your Language Models Secretly Decide How to Parallelize and Merge Generationcited by 30
  5. 2025Set Block Decoding is a Language Model Inference Acceleratorcited by 23
  6. 2025FS-DFM: Fast and Accurate Long Text Generation with Few-Step Diffusion Language Modelscited by 17
  7. 2025Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuningcited by 8
  8. 2025DiffuMamba: High-Throughput Diffusion LMs with Mamba Backbonecited by 6
  9. 2026DARE: Diffusion Large Language Models Alignment and Reinforcement Executorcited by 5
  10. 2026Learning from the Self-future: On-policy Self-distillation for dLLMscited by 5
  11. 2025BiTrajDiff: Bidirectional Trajectory Generation with Diffusion Models for Offline Reinforcement Learningcited by 4
  12. 2026ECHO: Efficient Chest X-ray Report Generation with One-step Block Diffusioncited by 4
5 more
  1. 2026Balancing Understanding and Generation in Discrete Diffusion Modelscited by 3
  2. 2026DICE: Diffusion Large Language Models Excel at Generating CUDA Kernelscited by 3
  3. 2026Gumbel Distillation for Parallel Text Generationcited by 1
  4. 2026Foundation Models for Medical Imaging: Status, Challenges, and Directionscited by 0
  5. 2026Destruction is a General Strategy to Learn Generation; Diffusion's Strength is to Take it Seriously; Exploration is the Futurecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025On Powerful Ways to Generate: Autoregression, Diffusion, and Beyond
  2. 2025Dream 7B: Diffusion Large Language Models
  3. 2025Esoteric Language Models: A Family of Any-Order Diffusion LLMs
  4. 2025Sequential Diffusion Language Models
  5. 2025FlashDLM: Accelerating Diffusion Language Model Inference via Efficient KV Caching and Guided Diffusion
  6. 2024Beyond Autoregression: Fast LLMs via Self-Distillation Through Time
  7. 2026Energy Consumption Analysis of Discrete Diffusion and Autoregressive Language Models
  8. 2025Theoretical Benefit and Limitation of Diffusion Language Model