mlsanityData to 7 Oct 2026

Sequential Diffusion Language Models

arXiv 2509.24007 ↗First posted 28 Sep 2025cs.CL cs.LGcited by 19

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Diffusion language models (DLMs) have strong theoretical efficiency but are limited by fixed-length decoding and incompatibility with key-value (KV) caches. Block diffusion mitigates these issues, yet still enforces a fixed block size and requires expensive training. We introduce Next Sequence Prediction (NSP), which unifies next-token and next-block prediction, enabling the model to adaptively determine the generation length at each step. When the length is fixed to 1, NSP reduces to standard next-token prediction. Building on NSP, we propose Sequential Diffusion Language Model (SDLM), which can retrofit pre-trained autoregressive language models (ALMs) at minimal cost. Specifically, SDLM performs diffusion inference within fixed-size mask blocks, but dynamically decodes consecutive subsequences based on model confidence, thereby preserving KV-cache compatibility and improving robustness to varying uncertainty and semantics across the sequence. Experiments show that SDLM matches or surpasses strong autoregressive baselines using only 3.5M training samples, while achieving 2.1 higher throughput than Qwen-2.5. Notably, the SDLM-32B model delivers even more pronounced efficiency gains, demonstrating the strong scalability potential of our modeling paradigm. Project page and codes: https://github.com/OpenGVLab/SDLM

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 43 of its 52 references that are in the map. First posted on arXiv 28 Sep 2025.

Builds on

43 of 52 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference4 Aug
  2. 2025dKV-Cache: The Cache for Diffusion Language Models21 May
  3. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models12 Mar
  4. 2025Large Language Diffusion Models14 Feb
  5. 2024Scaling up Masked Diffusion Models on Text24 Oct
  6. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Models23 Oct
  7. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data6 Jun
  8. 2023A Reparameterized Discrete Diffusion Model for Text Generation11 Feb

Other references · 35

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2025Qwen3 Technical Reportcited by 8,195
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  10. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  11. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  12. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
23 more
  1. 2021Program Synthesis with Large Language Modelscited by 3,509
  2. 2023Qwen Technical Reportcited by 3,273
  3. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  4. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  5. 2024Qwen2 Technical Reportcited by 2,076
  6. 2025Gemma 3 Technical Reportcited by 1,686
  7. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  8. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  9. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  10. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  11. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 540
  12. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 404
  13. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 344
  14. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  15. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 238
  16. 2024Flex Attention: A Programming Model for Generating Optimized Attention Kernelscited by 180
  17. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generationcited by 169
  18. 2025dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Cachingcited by 168
  19. 2024OpenCoder: The Open Cookbook for Top-Tier Code Large Language Modelscited by 127
  20. 2022DALL-E-Bot: Introducing Web-Scale Diffusion Models to Roboticscited by 105
  21. 2023Table-GPT: Table-tuned GPT for Diverse Table Taskscited by 69
  22. 2025Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potentialcited by 46
  23. 2024SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literaturecited by 37

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 15.

19 papers.

Sort:

By area:

  1. 2025WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inferencecited by 38
  2. 2025TiDAR: Think in Diffusion, Talk in Autoregressioncited by 35
  3. 2026LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decodingcited by 20
  4. 2026CoDAR: Continuous Diffusion Language Models are More Powerful Than You Thinkcited by 13
  5. 2026Stable-DiffCoder: Pushing the Frontier of Code Diffusion Large Language Modelcited by 12
  6. 2026Deferred Commitment Decoding for Diffusion Language Modelscited by 11
  7. 2025DiffusionVL: Translating Any Autoregressive Models into Diffusion Vision Language Modelscited by 10
  8. 2026Window-Diffusion: Accelerating Diffusion Language Model Inference with Windowed Token Pruning and Cachingcited by 2
  9. 2026Dataset-Level Metrics Attenuate Non-Determinism: A Fine-Grained Non-Determinism Evaluation in Diffusion Language Modelscited by 2
  10. 2026Towards A Generative Protein Evolution Machine with DPLM-Evocited by 2
  11. 2026Diffusion Language Models: An Experimental Analysiscited by 2
  12. 2026Set Diffusion: Interpolating Token Orderings Between Autoregression and Diffusion for Fast and Flexible Decodingcited by 2
7 more
  1. 2025Fast SceneScript: Fast and Accurate Language-Based 3D Scene Understanding via Multi-Token Predictioncited by 1
  2. 2026Reinforced Fast Weights with Next-Sequence Predictioncited by 1
  3. 2026Differences in Text Generated by Diffusion and Autoregressive Language Modelscited by 1
  4. 2026Bifocal Diffusion Language Models: Asymmetric Bidirectional Context for Parallel Generationcited by 0
  5. 2026Locate Anything in Videos: Rethinking Efficient Generative Spatio-Temporal Video Groundingcited by 0
  6. 2026d-OPD: Future-Aware On-Policy Distillation for Block Diffusion Language Modelscited by 0
  7. 2026HyperZip: Efficient Data Compression through Personalized Diffusion LLMs with Hypernetworkscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025FlashDLM: Accelerating Diffusion Language Model Inference via Efficient KV Caching and Guided Diffusion
  2. 2025A Survey on Diffusion Language Models
  3. 2025Diffusion Language Models Know the Answer Before Decoding
  4. 2025Diffusion Language Model Inference with Monte Carlo Tree Search
  5. 2025Fast-dLLM v2: Efficient Block-Diffusion LLM
  6. 2026CAI-DLLM: Convergence Aware Inference for Diffusion Language Models
  7. 2025Esoteric Language Models: A Family of Any-Order Diffusion LLMs
  8. 2025Beyond Next-Token Prediction: A Performance Characterization of Diffusion versus Autoregressive Language Models