mlsanityData to 14 Oct 2026

Set Block Decoding is a Language Model Inference Accelerator

arXiv 2509.04185 ↗First posted 4 Sep 2025cs.LGcited by 23

Not on the map. Why?

Abstract (arXiv)

Autoregressive next token prediction language models offer powerful capabilities but face significant challenges in practical deployment due to the high computational and memory costs of inference, particularly during the decoding stage. We introduce Set Block Decoding (SBD), a simple and flexible paradigm that accelerates generation by integrating standard next token prediction (NTP) and masked token prediction (MATP) within a single architecture. SBD allows the model to sample multiple, not necessarily consecutive, future tokens in parallel, a key distinction from previous acceleration methods. This flexibility allows the use of advanced solvers from the discrete diffusion literature, offering significant speedups without sacrificing accuracy. SBD requires no architectural changes or extra training hyperparameters, maintains compatibility with exact KV-caching, and can be implemented by fine-tuning existing next token prediction models. By fine-tuning Llama-3.1 8B and Qwen-3 8B, we demonstrate that SBD enables a 3-5x reduction in the number of forward passes required for generation while achieving same performance as equivalent NTP training.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Builds on 45 of its 53 references that are in the map. First posted on arXiv 4 Sep 2025.

Builds on

45 of 53 references.

References · 45

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  5. 2023Let's Verify Step by Stepcited by 3,789
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  7. 2021Program Synthesis with Large Language Modelscited by 3,637
  8. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  9. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  10. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  11. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  12. 2025Large Language Diffusion Modelscited by 912
33 more
  1. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  2. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 745
  3. 2024Simplified and Generalized Masked Diffusion for Discrete Datacited by 626
  4. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 578
  5. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 420
  6. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Modelscited by 409
  7. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 398
  8. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 360
  9. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 323
  10. 2024Discrete Flow Matchingcited by 317
  11. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 250
  12. 2025Mercury: Ultra-Fast Language Models Based on Diffusioncited by 235
  13. 2021Autoregressive Diffusion Modelscited by 210
  14. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generationcited by 187
  15. 2024Flex Attention: A Programming Model for Generating Optimized Attention Kernelscited by 185
  16. 2025Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inferencecited by 178
  17. 2025AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning datasetcited by 174
  18. 2025dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Cachingcited by 169
  19. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 148
  20. 2025dKV-Cache: The Cache for Diffusion Language Modelscited by 143
  21. 2025Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmaskingcited by 131
  22. 2025OpenCodeReasoning: Advancing Data Distillation for Competitive Codingcited by 113
  23. 2025Accelerating Diffusion LLMs via Adaptive Parallel Decodingcited by 112
  24. 2025Dimple: Discrete Diffusion Multimodal Large Language Model with Parallel Decodingcited by 101
  25. 2025FlashDLM: Accelerating Diffusion Language Model Inference via Efficient KV Caching and Guided Diffusioncited by 77
  26. 2013A Deep and Tractable Density Estimatorcited by 68
  27. 2024Speculative Diffusion Decoding: Accelerating Language Generation through Diffusioncited by 57
  28. 2025LongLLaDA: Unlocking Long Context Capabilities in Diffusion LLMscited by 40
  29. 2025Esoteric Language Models: A Family of Any-Order Diffusion LLMscited by 34
  30. 2024σ-GPTs: A New Approach to Autoregressive Modelscited by 20
  31. 2025Unifying Autoregressive and Diffusion-Based Sequence Generationcited by 17
  32. 2025CtrlDiff: Boosting Large Diffusion Language Models with Dynamic Block Prediction and Controllable Generationcited by 17
  33. 2025Reviving Any-Subset Autoregressive Models with Principled Parallel Sampling and Speculative Decodingcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 16.

23 papers.

Sort:

By area:

  1. 2025Fast-dLLM v2: Efficient Block-Diffusion LLMcited by 139
  2. 2025WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inferencecited by 41
  3. 2025Don't Settle Too Early: Self-Reflective Remasking for Diffusion Language Modelscited by 37
  4. 2025TiDAR: Think in Diffusion, Talk in Autoregressioncited by 37
  5. 2025Encoder-Decoder Diffusion Language Models for Efficient Training and Inferencecited by 24
  6. 2026Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decodingcited by 24
  7. 2025Dream-VL & Dream-VLA: Open Vision-Language and Vision-Language-Action Models with Diffusion Language Model Backbonecited by 18
  8. 2025OneFlow: Concurrent Mixed-Modal and Interleaved Generation with Edit Flowscited by 17
  9. 2026Fast-dVLM: Efficient Block-Diffusion VLM via Direct Conversion from Autoregressive VLMcited by 9
  10. 2026Introspective Diffusion Language Modelscited by 8
  11. 2026FBS: Modeling Native Parallel Reading inside a Transformercited by 7
  12. 2026Dynamic Expert Sharing: Decoupling Memory from Parallelism in Mixture-of-Experts Diffusion LLMscited by 4
11 more
  1. 2026Discrete Diffusion Models Exploit Asymmetry to Solve Lookahead Planning Taskscited by 4
  2. 2026Orthrus: Memory-Efficient Parallel Token Generation via Dual-View Diffusioncited by 4
  3. 2026dTRPO: Trajectory Reduction in Policy Optimization of Diffusion Large Language Modelscited by 3
  4. 2026SpecBlock: Block-Iterative Speculative Decoding with Dynamic Tree Draftingcited by 3
  5. 2026Fast Byte Latent Transformercited by 3
  6. 2026Unlocking Lossless Speedups in LLMs via Discrete Diffusioncited by 2
  7. 2026Accelerating Masked Diffusion Large Language Models: A Survey of Efficient Inference Techniquescited by 1
  8. 2026Context-weighted Discrete Flow Matchingcited by 0
  9. 2026Clock Diffusion: Efficient Semi-Autoregressive Continuous Diffusion Language Modelscited by 0
  10. 2026ALoDLM: Adaptively Looped Diffusion Language Modelscited by 0
  11. 2026DiffuPlex: Accelerating Full-Duplex Spoken Dialog Models via Rolling Masked Diffusioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025FastMTP: Accelerating LLM Inference with Enhanced Multi-Token Prediction
  2. 2026Multi-Token Prediction via Self-Distillation
  3. 2026Set Diffusion: Interpolating Token Orderings Between Autoregression and Diffusion for Fast and Flexible Decoding
  4. 2025Beyond Next-Token Prediction: A Performance Characterization of Diffusion versus Autoregressive Language Models
  5. 2026AngelSpec: Towards Real-World High Performance Inference with Speculative Decoding
  6. 2024Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decoding
  7. 2026MARS: Enabling Autoregressive Models Multi-Token Generation
  8. 2025Parallel Token Prediction for Language Models