mlsanityData to 14 Oct 2026

Adaptive DHiSparse: Speculative Decoding over a Load-Adaptive KV Hierarchy

arXiv 2610.13708 ↗First posted 9 Oct 2026cs.AI cs.CL cs.DCcited by 0

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Long-context serving runs out of KV memory before the hardware reaches its useful throughput. HiSparse stores history in host memory and fetches only the selected entries, but it uses one cache budget at every load and does not handle draft-model speculative decoding such as DSpark and DFlash2. We present Adaptive DHiSparse, which supports speculative decoding over hierarchical sparse KV and adapts residency to load. It protects the required working set, retains additional history while GPU memory is spare, and recovers the rest through exact fused retrieval. DSpark and DFlash2 keep their proposal and acceptance rules: queries share missing-row retrieval, draft KV uses a window-bounded ring, and verification staging is reused across layer groups. At 100K context and concurrency 128, GLM-5.3 with DFlash2 delivers 793 output tokens/s per decode GPU, 1.45× HiSparse and 1.27× non-speculative adaptive serving. Held to the same latency bounds as HiSparse, it sustains up to 4× the concurrency and 2.3 to 3.7× the output, depending on the per-token bound. On DeepSeek-V4.1-Flash at 500K context and concurrency 512, Adaptive DHiSparse with DSpark delivers 1.85× the output of resident serving with DSpark. In a load step, optional residency reduces host fetches from 14.2% to 1.4% of selected rows at low load and releases memory under pressure.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 21 of its 24 references that are in the map. First posted on arXiv 9 Oct 2026.

Builds on

21 of 24 references.

Closest ancestors · 9

Same direction, newest first

  1. 2026DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation6 Jul
  2. 2026DFlash: Block Diffusion for Flash Speculative Decoding5 Feb
  3. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test3 Mar
  4. 2024MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding20 Aug
  5. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees24 Jun
  6. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding18 Apr
  7. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  8. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  9. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 12

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  3. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 815
  4. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 245
  5. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 144
  6. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 134
  7. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attentioncited by 131
  8. 2026DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compressioncited by 76
  9. 2026IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reusecited by 31
  10. 2025Strata: Hierarchical Context Caching for Long Context Language Model Servingcited by 26
  11. 2026HiSparse: Scaling Sparse-Attention Decoding with Hierarchical KV Cache Managementcited by 3
  12. 2026DualDecoder: Accelerate Long Context LLM Inference by Predictive Prefetchcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Nightjar: Dynamic Adaptive Speculative Decoding for Large Language Models Serving
  2. 2026DScale: Scaling Block-Diffusion Speculative Decoding with Adaptive Verification
  3. 2026CATS: Cascaded Adaptive Tree Speculation for Memory-Limited LLM Inference Acceleration
  4. 2026Bastion: Budget-Aware Speculative Decoding with Tree-structured Block Diffusion Drafting
  5. 2024CREST: Effectively Compacting a Datastore For Retrieval-Based Speculative Decoding
  6. 2026ECHO: Elastic Speculative Decoding with Sparse Gating for High-Concurrency Scenarios
  7. 2026A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding
  8. 2025SPIRe: Boosting LLM Inference Throughput with Speculative Decoding