mlsanityData to 14 Oct 2026

FOCUS: DLLMs Know How to Tame Their Compute Bound

arXiv 2601.23278 ↗First posted 30 Jan 2026cs.LG cs.AR cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

Diffusion Large Language Models (DLLMs) offer a compelling alternative to Auto-Regressive models, but their deployment is constrained by high decoding cost. In this work, we identify a key inefficiency in DLLM decoding: while computation is parallelized over token blocks, only a small subset of tokens is decodable at each diffusion step, causing most compute to be wasted on non-decodable tokens. We further observe a strong correlation between attention-derived token importance and token-wise decoding probability. Based on this insight, we propose FOCUS, an inference system designed for DLLMs. By dynamically focusing computation on decodable tokens and evicting non-decodable ones on-the-fly, FOCUS increases the effective batch size, alleviating compute limitations and enabling scalable throughput. Empirical evaluations demonstrate that FOCUS achieves up to 3.52× throughput improvement over the production-grade engine LMDeploy in large-batch settings, while preserving or improving generation quality across multiple benchmarks.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 36 of its 49 references that are in the map. First posted on arXiv 30 Jan 2026.

Builds on

36 of 49 references.

References · 36

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2021Program Synthesis with Large Language Modelscited by 3,637
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  7. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  9. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  10. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  11. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,027
  12. 2025Large Language Diffusion Modelscited by 912
24 more
  1. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  2. 2024Simple and Effective Masked Diffusion Language Modelscited by 848
  3. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  4. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 745
  5. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
  6. 2025Dream 7B: Diffusion Large Language Modelscited by 544
  7. 2022Efficiently Scaling Transformer Inferencecited by 539
  8. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
  9. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 432
  10. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 420
  11. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Modelscited by 409
  12. 2025dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Cachingcited by 169
  13. 2025LLaDA2.0: Scaling Up Diffusion Language Models to 100Bcited by 166
  14. 2025dKV-Cache: The Cache for Diffusion Language Modelscited by 143
  15. 2025Fast-dLLM v2: Efficient Block-Diffusion LLMcited by 139
  16. 2025SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generationcited by 129
  17. 2025Sparse-dLLM: Accelerating Diffusion LLMs with Dynamic Cache Evictioncited by 65
  18. 2025dParallel: Learnable Parallel Decoding for dLLMscited by 65
  19. 2025Diffusion Language Models are Super Data Learnerscited by 48
  20. 2025WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inferencecited by 41
  21. 2025Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in Speedcited by 33
  22. 2025Learning to Parallel: Accelerating Diffusion Large Language Models via Learnable Parallel Decodingcited by 27
  23. 2025d2Cache: Accelerating Diffusion-Based LLMs via Dual Adaptive Cachingcited by 25
  24. 2025How Efficient Are Diffusion Language Models? A Critical Examination of Efficiency Evaluation Practicescited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026CORA-Diff: Confidence-Oriented Residual Acceptance for Efficient Diffusion Language Model Inferencecited by 0
  2. 2026Affix Cache for Diffusion Large Language Modelscited by 0
  3. 2026FastGuide: Accelerating Reward Guidance for Diffusion Large Language Modelscited by 0