mlsanityData to 14 Oct 2026

Q-First: Most of Attention Needs Only the Query in Disaggregated LLM Decoding

arXiv 2608.15473 ↗First posted 16 Aug 2026cs.DCcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Disaggregated LLM serving puts the KV-cache sweep on memory-optimised hardware and the projections and feed-forward on compute-optimised hardware, then inherits from the decoder block a dependency neither device wants: attention runs first and the feed-forward consumes its output, so within one sequence each side idles while the other works. The usual repair costs one resident KV cache per extra sequence in flight, which is what motivated separating the devices at all. We remove the dependency instead. The sweep needs only the query, and exchanging the two sub-layers makes that query available while the compute side still has work to do, so the two run concurrently; the current key and value follow as a cache write nothing waits on. We state the decode as a protocol, show that it runs on stock kernels, and verify it end to end on a trained checkpoint to a relative error of 3.2x10^-3 -- with no new operator, no changed shape and no new hardware. We then train the block 8 ways at two seeds each, varying only where the attention reads and holding everything else fixed. At three per cent of compute-optimal a lead in bits per byte measures how much a change disturbed training rather than what it reaches, so we read magnitudes and not rankings. Among the 5 blocks whose feed-forward does not consume their own attention, no read point differs from the one that moves nothing by more than 0.0026 bits per byte -- smaller than the gap between an arm and itself at a second seed, 0.0066 -- while the same runs resolve a sub-layer exchange 25 times as large. Moving the query early is a change the measurement cannot find, which is what the protocol needs. The reach is bounded: projecting every layer's query from the network's input costs +0.0974, refuting a pre-registered threshold at both seeds, so a query may be read one feed-forward early and no further back.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 17 of its 19 references that are in the map. First posted on arXiv 16 Aug 2026.

Builds on

17 of 19 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026Move the Query, Not the Cache: Characterizing Cross-Instance Latent Attention Redistribution Across GPU Fabrics31 May
  2. 2024InstInfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inference8 Sep
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 14

  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  3. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  4. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  5. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  6. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  7. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  8. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  9. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  10. 2018Online normalizer calculation for softmaxcited by 162
  11. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCachecited by 73
  12. 2025Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decodingcited by 55
2 more
  1. 2025MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelismcited by 32
  2. 2025Injecting Adrenaline into LLM Serving: Boosting Resource Utilization and Throughput via Attention Disaggregationcited by 14

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Analytical Provisioning for Attention-FFN Disaggregated LLM Serving under Stochastic Workloads
  2. 2025KV Cache Recycling to Expand Usable Context Capacity in Low Parameter LLMs
  3. 2026SAC: Disaggregated KV Cache System for Sparse Attention LLMs with CXL
  4. 2026Where Activation Sparsity and KV-Cache Sparsity Cross in LLM Decoding
  5. 2025PAT: Accelerating LLM Decoding via Prefix-Aware Attention with Resource Efficient Multi-Tile Kernel
  6. 2026OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetching
  7. 2025TraCT: Disaggregated LLM Serving with CXL Shared Memory KV Cache at Rack-Scale
  8. 2026HiKV: Hierarchical Importance-Aware KV Cache with Hardware Acceleration for LLM Decoding