mlsanityData to 7 Oct 2026

SPA-Cache: Singular Proxies for Adaptive Caching in Diffusion Language Models

arXiv 2602.02544 ↗First posted 30 Jan 2026cs.LG cs.AIcited by 1

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

While Diffusion Language Models (DLMs) offer a flexible, arbitrary-order alternative to the autoregressive paradigm, their non-causal nature precludes standard KV caching, forcing costly hidden state recomputation at every decoding step. Existing DLM caching approaches reduce this cost by selective hidden state updates; however, they are still limited by (i) costly token-wise update identification heuristics and (ii) rigid, uniform budget allocation that fails to account for heterogeneous hidden state dynamics. To address these challenges, we present SPA-Cache that jointly optimizes update identification and budget allocation in DLM cache. First, we derive a low-dimensional singular proxy that enables the identification of update-critical tokens in a low-dimensional subspace, substantially reducing the overhead of update identification. Second, we introduce an adaptive strategy that allocates fewer updates to stable layers without degrading generation quality. Together, these contributions significantly improve the efficiency of DLMs, yielding up to an 8× throughput improvement over vanilla decoding and a 2--4× speedup over existing caching baselines.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 52 of its 58 references that are in the map. First posted on arXiv 30 Jan 2026.

Builds on

52 of 58 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025Dream 7B: Diffusion Large Language Models21 Aug
  2. 2025Time Is a Feature: Exploiting Temporal Dynamics in Diffusion Language Models12 Aug
  3. 2025Diffusion LLMs Can Do Faster-Than-AR Inference via Discrete Diffusion Forcing8 Aug
  4. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation25 Jun
  5. 2025dKV-Cache: The Cache for Diffusion Language Models21 May
  6. 2025Large Language Diffusion Models14 Feb
  7. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  8. 2023Likelihood-Based Diffusion Language Models30 May
  9. 2022Continuous diffusion for categorical data28 Nov
  10. 2022DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models17 Oct

Other references · 42

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2016Layer Normalizationcited by 5,579
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2021Program Synthesis with Large Language Modelscited by 3,509
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  8. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  9. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  10. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  11. 2019Root Mean Square Layer Normalizationcited by 1,425
  12. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
30 more
  1. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  2. 2020On Layer Normalization in the Transformer Architecturecited by 906
  3. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  4. 2022Efficiently Scaling Transformer Inferencecited by 529
  5. 2019Scaling Out-of-Distribution Detection for Real-World Settingscited by 465
  6. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 404
  7. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
  8. 2018Regularisation of Neural Networks by Enforcing Lipschitz Continuitycited by 311
  9. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 273
  10. 2019Representation Degeneration Problem in Training Natural Language Generation Modelscited by 250
  11. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 241
  12. 2025Mercury: Ultra-Fast Language Models Based on Diffusioncited by 225
  13. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learningcited by 178
  14. 2025dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Cachingcited by 168
  15. 2021All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Qualitycited by 150
  16. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 145
  17. 2025Fast-dLLM v2: Efficient Block-Diffusion LLMcited by 128
  18. 2025Accelerating Diffusion LLMs via Adaptive Parallel Decodingcited by 108
  19. 2025Sparse-dLLM: Accelerating Diffusion LLMs with Dynamic Cache Evictioncited by 64
  20. 2025Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectorscited by 59
  21. 2025SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Datacited by 50
  22. 2025Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoningcited by 43
  23. 2025A Survey of Direct Preference Optimizationcited by 35
  24. 2025Attention Is All You Need for KV Cache in Diffusion LLMscited by 32
  25. 2023Sequence Length Independent Norm-Based Generalization Bounds for Transformerscited by 28
  26. 2025d2Cache: Accelerating Diffusion-Based LLMs via Dual Adaptive Cachingcited by 24
  27. 2025VORTA: Efficient Video Diffusion via Routing Sparse Attentioncited by 23
  28. 2024AsymRnR: Video Diffusion Transformers Acceleration with Asymmetric Reduction and Restorationcited by 15
  29. 2025Mixture of Experts in Large Language Modelscited by 15
  30. 2025Fast and Accurate Causal Parallel Decoding using Jacobi Forcingcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026SpecFold: Folding Multi-Branch Redundancy for Faster Speculative Decoding in Diffusion Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Archer: Adaptive Reuse of Cached Hidden States for Efficient Rollback in Diffusion Language Models
  2. 2026EntropyCache: Decoded Token Entropy Guided KV Caching for Diffusion Language Models
  3. 2026Polestar: Drift-Aware Cache Calibration and Token Commitment for Efficient Inference of Diffusion LLMs
  4. 2025FlashDLM: Accelerating Diffusion Language Model Inference via Efficient KV Caching and Guided Diffusion
  5. 2026Fixed State, Long Reach: What a Constant-Size Cache Buys Block Diffusion at Scale
  6. 2026Affix Cache for Diffusion Large Language Models
  7. 2026Accelerating Masked Diffusion Large Language Models: A Survey of Efficient Inference Techniques
  8. 2025Esoteric Language Models: A Family of Any-Order Diffusion LLMs