mlsanityData to 14 Oct 2026

KV Prediction for Improved Time to First Token

arXiv 2410.08391 ↗First posted 10 Oct 2024cs.CL cs.AIcited by 11

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Inference with transformer-based language models begins with a prompt processing step. In this step, the model generates the first output token and stores the KV cache needed for future generation steps. This prompt processing step can be computationally expensive, taking 10s of seconds or more for billion-parameter models on edge devices when prompt lengths or batch sizes rise. This degrades user experience by introducing significant latency into the model's outputs. To reduce the time spent producing the first output (known as the ``time to first token'', or TTFT) of a pretrained model, we introduce a novel method called KV Prediction. In our method, a small auxiliary model is used to process the prompt and produce an approximation of the KV cache used by a base model. This approximated KV cache is then used with the base model for autoregressive generation without the need to query the auxiliary model again. We demonstrate that our method produces a pareto-optimal efficiency-accuracy trade-off when compared to baselines. On TriviaQA, we demonstrate relative accuracy improvements in the range of 15%-50% across a range of TTFT FLOPs budgets. We also demonstrate accuracy improvements of up to 30% on HumanEval python code completion at fixed TTFT FLOPs budgets. Additionally, we benchmark models on an Apple M2 Pro CPU and demonstrate that our improvement in FLOPs translates to a TTFT speedup on hardware. We release our code at https://github.com/apple/corenet/tree/main/projects/kv-prediction .

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 39 of its 43 references that are in the map. First posted on arXiv 10 Oct 2024.

Builds on

39 of 43 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  2. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attention21 May
  3. 2024Layer-Condensed KV Cache for Efficient Inference of Large Language Models17 May
  4. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  5. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct

Other references · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  6. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  7. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  8. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  9. 2020Knowledge Distillation: A Surveycited by 1,603
  10. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  11. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  12. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
22 more
  1. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  2. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 453
  3. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  4. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 378
  5. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 367
  6. 2022The Stack: 3 TB of permissively licensed source codecited by 359
  7. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 287
  8. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  9. 2020Query-Key Normalization for Transformerscited by 280
  10. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 232
  11. 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 199
  12. 2023Compressing Context to Enhance Inference Efficiency of Large Language Modelscited by 189
  13. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 177
  14. 2019On the Use of ArXiv as a Datasetcited by 103
  15. 2024OpenELM: An Efficient Language Model Family with Open Training and Inference Frameworkcited by 75
  16. 2024LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inferencecited by 70
  17. 2018Label Refinery: Improving ImageNet Classification through Label Progressioncited by 65
  18. 2024Towards Greener LLMs: Bringing Energy-Efficiency to the Forefront of LLM Inferencecited by 61
  19. 2024Learn To be Efficient: Build Structured Sparsity in Large Language Modelscited by 20
  20. 2022Random-LTD: Random and Layerwise Token Dropping Brings Efficient Training for Large-scale Transformerscited by 11
  21. 2024Tandem Transformers for Inference Efficient LLMscited by 10
  22. 2024CritiPrefill: A Segment-wise Criticality-based Approach for Prefilling Acceleration in LLMscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 6.

11 papers.

  1. 2025QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cachecited by 27
  2. 2025Speculative Prefill: Turbocharging TTFT with Lightweight and Training-Free Token Importance Estimationcited by 14
  3. 2026Scheduling LLM Inference with Uncertainty-Aware Output Length Predictionscited by 8
  4. 2026When to Think, When to Speak: Learning Disclosure Policies for LLM Reasoningcited by 5
  5. 2025Titanus: Enabling KV Cache Pruning and Quantization On-the-Fly for LLM Accelerationcited by 2
  6. 2025Don't Pay Attentioncited by 1
  7. 2026KV-Lingo: Learning KV-Cache Translators with Distillationcited by 1
  8. 2025Make a Video Call with LLM: A Measurement Campaign over Six Mainstream Appscited by 0
  9. 2026Non-Monotonic Latency in Apple MPS Decoding: KV Cache Interactions and Execution Regimescited by 0
  10. 2026Just MLPs: Efficient Visual State Reconstruction for Multimodal Language Modelscited by 0
  11. 2026Towards Looped Models Done Right, Part II: Rethinking at Fixed Pointscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generation
  2. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  3. 2024RefreshKV: Updating Small KV Cache During Long-form Generation
  4. 2024KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing
  5. 2026Comparative Characterization of KV Cache Management Strategies for LLM Inference
  6. 2024A Survey on Large Language Model Acceleration based on KV Cache Management
  7. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference
  8. 2025CriticalKV: Optimizing KV Cache Eviction from an Output Perturbation Perspective