mlsanityData to 14 Oct 2026

Shallow Prefill, Deep Decoding: Efficient Long-Context Inference via Layer-Asymmetric KV Visibility

arXiv 2605.06105 ↗First posted 7 May 2026cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Long-context inference in decoder-only language models is costly because long prompts are processed during Prefill, cached at every layer, and repeatedly attended to during autoregressive Decode. We introduce Shallow Prefill, dEEp Decode (SPEED), a phase-asymmetric KV-visibility policy that materializes non-anchor prompt-token KV states only in lower layers while keeping Decode-phase tokens full-depth. Unlike previous approaches that make upper-layer prompt KV states cheaper to store or construct, SPEED removes prefill tokens from the upper-layer Decode visibility set altogether. With a minimal BoS anchor, this simple change preserves broad benchmark quality while reducing long-context cost. In a controlled Llama-3.1-8B instruction-tuning study, SPEED using only 75% of layers for prefill tokens reaches 51.2 average score on OLMES-style benchmarks, compared with 51.4 for the full-depth baseline, while improving TTFT by 33%, TPOT by 22%, and reducing active KV memory by 25.0% at 128K context. Layer-wise diagnostics suggest that this cutoff retains the main prompt-selection and representation-stabilization regions of the full-depth model. These results show that long-context prompt tokens need not always persist as full-depth KV-cache objects when Decode-phase tokens remain full-depth.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 36 of its 37 references that are in the map. First posted on arXiv 7 May 2026.

Builds on

36 of 37 references.

References · 36

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  6. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  7. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  8. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  9. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  10. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  11. 2022Efficiently Scaling Transformer Inferencecited by 539
  12. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
24 more
  1. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 439
  2. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 432
  3. 2023Learning to Compress Prompts with Gist Tokenscited by 390
  4. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  5. 2023Adapting Language Models to Compress Contextscited by 356
  6. 2022Confident Adaptive Language Modelingcited by 310
  7. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 307
  8. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 288
  9. 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 199
  10. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 144
  11. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attentioncited by 131
  12. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Modelscited by 125
  13. 2024OLMES: A Standard for Language Model Evaluationscited by 106
  14. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttentioncited by 98
  15. 2024What Matters in Transformers? Not All Attention is Neededcited by 84
  16. 2024Layer-Condensed KV Cache for Efficient Inference of Large Language Modelscited by 59
  17. 2023Large language models implicitly learn to straighten neural sentence trajectories to construct a predictive representation of natural languagecited by 32
  18. 2024Attend First, Consolidate Later: On the Importance of Attention in Different LLM Layerscited by 25
  19. 2024SwiftKV: Fast Prefill-Optimized Inference with Knowledge-Preserving Model Transformationcited by 19
  20. 2025KV Admission: Learning What to Write for Efficient Long-Context LLM Inferencecited by 4
  21. 2026DepthKV: Layer-Dependent KV Cache Pruning for Long-Context LLM Inferencecited by 3
  22. 2025Data-Free Pruning of Self-Attention Layers in LLMscited by 2
  23. 2026POP: Prefill-Only Pruning for Efficient Large Model Inferencecited by 2
  24. 2025High-Layer Attention Pruning with Rescalingcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026ReCache: Efficient KV Cache Reuse and Compression for Tool-Augmented LLM Agentscited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2026UniPrefill: Universal Long-Context Prefill Acceleration via Block-wise Dynamic Sparsification
  2. 2026A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding
  3. 2025FastKV: Decoupling of Context Reduction and KV Cache Compression for Prefill-Decoding Acceleration
  4. 2026Lynx: Progressive Speculative Quantization for accelerating KV Transfer in Long-Context Inference
  5. 2025Efficient Pretraining Length Scaling
  6. 2026DualDecoder: Accelerate Long Context LLM Inference by Predictive Prefetch
  7. 2025KV Pareto: Systems-Level Optimization of KV Cache and Model Compression for Long Context Inference
  8. 2024Accelerating Large Language Model Inference with Self-Supervised Early Exits