mlsanityData to 14 Oct 2026

GroupKV: Hierarchical KV Cache Management for Long-Context Diffusion LLM Inference

arXiv 2609.17573 ↗First posted 30 Jul 2026cs.OS cs.DC cs.LGcited by 2

On the map: Efficient LLM inference and serving → Diffusion LLM and MoE inference

Abstract (arXiv)

Diffusion large language models (dLLMs) are emerging as a promising generative paradigm that complements autoregressive decoding. In long-context settings, KV cache bloat and offloading transfer overhead have become primary bottlenecks in inference systems. Meanwhile, the periodic full-sequence recomputation and localized token updates in dLLMs make the KV lifecycle substantially more dynamic, complicating cache management and prefetch scheduling while making heavyweight token-level indexing or clustering schemes harder to amortize effectively during decoding. To address these challenges, we present textscGroupKV, a lightweight hierarchical KV cache management system for long-context dLLM inference. We observe that under block-wise decoding, tokens within the same generation block tend to access highly overlapping and spatially concentrated context regions, making group-level sparse selection effective. Building on this observation, textscGroupKV partitions the context into contiguous groups and performs coarse-to-fine sparse selection. textscGroupKV further exploits cross-layer consistency to enable predictive prefetching, and incorporates a staleness correction mechanism to maintain cache coherence under dynamic KV updates. Additionally, textscGroupKV adopts streaming prefill to reduce peak memory consumption during prefilling. Experiments show that textscGroupKV extends the maximum serviceable context length by up to 48.00× under constrained GPU memory, improves end-to-end inference performance by up to 3.73× in offload-based long-context settings, and maintains competitive task accuracy.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Diffusion LLM and MoE inference, one of 12 directions in Efficient LLM inference and serving. Builds on 33 of its 45 references that are in the map. First posted on arXiv 30 Jul 2026.

Builds on

33 of 45 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Attention Is All You Need for KV Cache in Diffusion LLMs16 Oct
  2. 2025Mask Tokens as Prophet: Fine-Grained Cache Eviction for Efficient dLLM Inference10 Oct
  3. 2025Diffusion LLMs Can Do Faster-Than-AR Inference via Discrete Diffusion Forcing8 Aug
  4. 2025Sparse-dLLM: Accelerating Diffusion LLMs with Dynamic Cache Eviction4 Aug
  5. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding28 May
  6. 2025dKV-Cache: The Cache for Diffusion Language Models21 May

Other references · 27

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  3. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  4. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  5. 2025Large Language Diffusion Modelscited by 912
  6. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  7. 2024Simple and Effective Masked Diffusion Language Modelscited by 848
  8. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  9. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 745
  10. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  11. 2025Dream 7B: Diffusion Large Language Modelscited by 544
  12. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
15 more
  1. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Modelscited by 409
  2. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  3. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 245
  4. 2025Mercury: Ultra-Fast Language Models Based on Diffusioncited by 235
  5. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  6. 2025dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Cachingcited by 169
  7. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 148
  8. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attentioncited by 131
  9. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 116
  10. 2025FlashDLM: Accelerating Diffusion Language Model Inference via Efficient KV Caching and Guided Diffusioncited by 77
  11. 2025LLaDA-MoE: A Sparse MoE Diffusion Language Modelcited by 57
  12. 2024FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelinescited by 49
  13. 2025LongLLaDA: Unlocking Long Context Capabilities in Diffusion LLMscited by 40
  14. 2025DPad: Efficient Diffusion Language Models with Suffix Dropoutcited by 39
  15. 2026Nemotron-Labs-TwoTower: Diffusion Language Modeling with Pretrained Autoregressive Contextcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026d-OPD: Future-Aware On-Policy Distillation for Block Diffusion Language Modelscited by 0
  2. 2026Mask-Guided KV Cache Eviction in Block Diffusion Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025d2Cache: Accelerating Diffusion-Based LLMs via Dual Adaptive Caching
  2. 2026Polestar: Drift-Aware Cache Calibration and Token Commitment for Efficient Inference of Diffusion LLMs
  3. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference
  4. 2025CentroidKV: Efficient Long-Context LLM Inference via KV Cache Clustering
  5. 2025WindowKV: Task-Adaptive Group-Wise KV Cache Window Selection for Efficient LLM Inference
  6. 2026Mosaic: Unlocking Long-Context Inference for Diffusion LLMs via Global Memory Planning and Dynamic Peak Taming
  7. 2026Accelerating Masked Diffusion Large Language Models: A Survey of Efficient Inference Techniques
  8. 2026Flash-dLLM: IO-Aware KV Caching and Parallel Decoding for Fast, Memory-Efficient Diffusion LLMs