mlsanityData to 14 Oct 2026

Fathom: Per-Query Read Depth for Sparse Decoding over Offloaded KV Caches

arXiv 2609.17652 ↗First posted 15 Sep 2026cs.LG cs.CL cs.DCcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

When agentic sessions run to a million tokens with many sessions resident at once, the KV cache and the index that ranks it live in host memory, and the scan that ranks all n keys for a top-k step becomes the traffic that bounds decoding. We present Fathom, a key scan in which each query decides how many bits of each key channel to read. The 4-bit K cache is stored channel-major as bit planes, so a prefix of t planes is exactly the channel's t-bit quantizer, and the query spends its bit budget by reverse water-filling over the variance-weighted importance of its channels. At one million tokens on Qwen3-8B a decode step is 1.67x faster in GPU time than with the 136-bit scans of Double Sparsity, Loki and SparQ r=32, and in the same GPU time as SparQ's 68-bit read (r=16) Fathom reads 18% fewer bytes with lower attention error on six of seven model and context settings. On RULER-style tasks every per-token scan matches exact top-k decoding, and on real coding-agent sessions Fathom reaches the step agreement of the most accurate 136-bit scan at 92 bits. The store is the 4-bit K copy a quantized serving stack already holds, and the method is not faster when the index is resident in GPU memory.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 24 of its 27 references that are in the map. First posted on arXiv 15 Sep 2026.

Builds on

24 of 27 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026Sparse Attention as a Range Searching Problem: Towards an Inference-Efficient Index for KV Cache7 May
  2. 2026Self-Indexing KVCache: Predicting Sparse Attention from Compressed Keys15 Mar
  3. 2025RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inference5 May
  4. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference28 Oct
  5. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  6. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  7. 2023SparQ Attention: Bandwidth-Efficient LLM Inference8 Dec

Other references · 17

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  4. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  5. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  6. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 1,002
  7. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  8. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  9. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 197
  10. 2025SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agentscited by 142
  11. 2025TurboQuant: Online Vector Quantization with Near-optimal Distortion Ratecited by 125
  12. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 116
5 more
  1. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 111
  2. 2024Loki: Low-rank Keys for Efficient Sparse Attentioncited by 89
  3. 2024Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMscited by 54
  4. 2024Post-Training Sparse Attention with Double Sparsitycited by 45
  5. 2026Faster Than Flash: Exploiting Attention Sparsity for Efficient Long-Context Decodingcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Read What Matters: Query-Adaptive Quantization for KV Caches
  2. 2026Where Activation Sparsity and KV-Cache Sparsity Cross in LLM Decoding
  3. 2026MOIRA: Mass-Oriented Indexing with Ragged Attention for Long-Context Decoding
  4. 2026Spend Bits Where Queries Look: KV Cache Vector Quantization with Attention-Preserving Transforms
  5. 2026CORE: COverage CAlibration and Evicted-Mass REdistribution for KV Cache
  6. 2026HiSparse: Scaling Sparse-Attention Decoding with Hierarchical KV Cache Management
  7. 2024HashEvict: A Pre-Attention KV Cache Eviction Strategy using Locality-Sensitive Hashing
  8. 2026HeadQ: Model-Visible Distortion and Score-Space Correction for KV-Cache Quantization