mlsanityData to 7 Oct 2026

RedKnot-MLA: Multi-Head Offline-Online Reuse for DeepSeek-V4 Long-Context Serving

arXiv 2609.07008 ↗First posted 7 Sep 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Multi-head latent attention (MLA) exposes many logical query heads through one packed latent KV stream. This representation is memory efficient, but it removes the physical per-head cache boundary assumed by conventional head-wise reuse. We present our system, a DeepSeek-V4 realization of RedKnot's head-aware reuse principle. Each immutable document is processed offline at canonical position zero; certified Local-head contributions are retained as MLA-Off. At serving time, query-side RoPE relocation restores the document's request position, a small Global-head set and protected Local token rows are recomputed as MLA-Online, and the two paths are merged before a single shared output projection. The packed MLA latent is never split. DeepSeek-V4-Flash uses 37 reusable layers and a 56/8 Local/Global partition, giving a 75.29% analytic logical head-row ceiling; the Pro-0813 profile uses 55 layers and 112/16 heads, giving 78.89%. Frozen Flash operating points show hot-artifact TTFT speedups of 2.02-3.84x. At 256K, the archived three-dataset study reports an aggregate F1 change of +3.24 percentage points, an EM change of +4.16 points, and a 78.7-79.5% analytic major-operator arithmetic saving, while one dataset decreases by 2.81 F1 points. A separate author-reported 256K hot-artifact QPS measurement is approximately 2.0x; because its raw concurrency trace is not included in this bundle, we mark it as preliminary rather than archived evidence. We describe the factorization, position repair, token-row closure, sparse-MoE support, TP8 integration, and the measurement boundaries needed to interpret these results.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 6 of its 7 references that are in the map. First posted on arXiv 7 Sep 2026.

Builds on

6 of 7 references.

References · 6

  1. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  2. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 877
  3. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 211
  4. 2024EPIC: Efficient Position-Independent Caching for Serving Large Language Modelscited by 45
  5. 2026HYPIC: Accelerating Hybrid-Attention LLM Serving with Position-Independent Cachingcited by 7
  6. 2026RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttentioncited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Hardware-Centric Analysis of DeepSeek's Multi-Head Latent Attention
  2. 2026Through the Bottleneck: How Multi-head Latent Attention Separates Content from Position in Language Models
  3. 2025Towards Economical Inference: Enabling DeepSeek's Multi-Head Latent Attention in Any Transformer-based LLMs
  4. 2025Latent Multi-Head Attention for Small Language Models
  5. 2026DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression
  6. 2025X-EcoMLA: Upcycling Pre-Trained Attention into MLA for Efficient and Extreme KV Compression
  7. 2026Beyond KV Reconstruction: Functional Reconstruction for MLA Draft Models in Speculative Decoding
  8. 2026MHA2MLA-VLM: Enabling DeepSeek's Economical Multi-Head Latent Attention across Vision-Language Models