mlsanityData to 14 Oct 2026

HERALD: High-Throughput Block Diffusion LLM Serving via CPU-GPU Cooperative KV Cache Retrieval

arXiv 2606.21633 ↗First posted 19 Jun 2026cs.LGcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

The KV cache dominates GPU memory in long-context LLM serving, crowding out batch capacity and leaving GPU compute idle. Offloading the cache to CPU DRAM restores capacity, but the limited PCIe bandwidth forces state-of-the-art offloading systems to pair it with sparse attention, fetching only a small critical subset of the cache to the GPU. These systems, however, follow the KV access pattern of autoregressive decoding, in which the critical set changes at every token: selection and fetching recur at every decoding step, and throughput remains capped by PCIe bandwidth rather than by either processor. Block diffusion LLMs(block dLLMs), which decode a block of B tokens over T denoising steps, exhibit a different KV access pattern that opens a new opportunity for offloading. Recent sparse block dLLM methods have shown that sparse inference separates into a selection phase that scans the full KV cache once per block and a denoising phase that reuses the selected small subset T times. This asymmetry aligns with the compute and memory asymmetry of a CPU-GPU system, making it advantageous to run selection on the CPU and denoising on the GPU: the critical KV cache then crosses PCIe only once per block, removing the interconnect as the bottleneck. We present HERALD, to our knowledge the first KV offloading system designed for block dLLMs. HERALD resolves the two obstacles of this mapping, the serialized dependency between the phases and the compute-bound B-query selection on the CPU, by overlapping the phases with a draft block, reducing the selection cost with a single [MASK] query, and executing both as a dual-stream pipeline over double-buffered sparse KV pools. On two production block dLLMs, HERALD sustains near-lossless accuracy at a 5% KV budget and reaches up to 2.28x the decode throughput of GPU-only serving, with gains that widen with context length.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 33 of its 40 references that are in the map. First posted on arXiv 19 Jun 2026.

Builds on

33 of 40 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference8 Oct
  2. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving2 Jan
  3. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference28 Oct
  4. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  5. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  6. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention23 Mar
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 26

  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  2. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  3. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  4. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  5. 2025Large Language Diffusion Modelscited by 912
  6. 2024Simple and Effective Masked Diffusion Language Modelscited by 848
  7. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 745
  8. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  9. 2025Dream 7B: Diffusion Large Language Modelscited by 544
  10. 2022DiffuSeq: Sequence to Sequence Text Generation with Diffusion Modelscited by 450
  11. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 420
  12. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Modelscited by 409
14 more
  1. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Modelscited by 274
  2. 2025Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusionscited by 214
  3. 2025dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Cachingcited by 169
  4. 2025LLaDA2.0: Scaling Up Diffusion Language Models to 100Bcited by 166
  5. 2025dKV-Cache: The Cache for Diffusion Language Modelscited by 143
  6. 2025Fast-dLLM v2: Efficient Block-Diffusion LLMcited by 139
  7. 2025SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generationcited by 129
  8. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 112
  9. 2025Sparse-dLLM: Accelerating Diffusion LLMs with Dynamic Cache Evictioncited by 65
  10. 2025Accelerating Diffusion Large Language Models with SlowFast Sampling: The Three Golden Principlescited by 45
  11. 2024NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inferencecited by 37
  12. 2025CDLM: Consistency Diffusion Language Models For Faster Samplingcited by 21
  13. 2025SparseD: Sparse Attention for Diffusion Language Modelscited by 17
  14. 2025SpeContext: Enabling Efficient Long-context Reasoning with Speculative Context Sparsity in LLMscited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026PulseInfer: I/O-Centric Sparse KV Cache Offloading for Efficient Long-Context LLM Decoding
  2. 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference
  3. 2026ScoutAttention: Efficient KV Cache Offloading via Layer-Ahead CPU Pre-computation for LLM Inference
  4. 2026Optimus: Elastic Decoding for Efficient Diffusion LLM Serving
  5. 2025Understanding Bottlenecks for Efficiently Serving LLM Inference With KV Offloading
  6. 2026Serving Masked Diffusion LLMs: Characterization and Design Principles from Real Hardware
  7. 2026Flash-dLLM: IO-Aware KV Caching and Parallel Decoding for Fast, Memory-Efficient Diffusion LLMs
  8. 2026PipeMax: Enhancing Offline LLM Inference on Commodity GPU Servers