mlsanityData to 14 Oct 2026

SwiftKV: Fast Prefill-Optimized Inference with Knowledge-Preserving Model Transformation

arXiv 2410.03960 ↗First posted 4 Oct 2024cs.LG cs.AI cs.CLcited by 19

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

LLM inference for enterprise applications, such as summarization, RAG, and code-generation, typically observe much longer prompt than generations, leading to high prefill cost and response latency. We present SwiftKV, a novel model transformation and distillation procedure targeted at reducing the prefill compute (in FLOPs) of prompt tokens while preserving high generation quality. First, SwiftKV prefills later layers' KV cache using an earlier layer's output, allowing prompt tokens to skip those later layers. Second, SwiftKV employs a lightweight knowledge-preserving distillation procedure that can adapt existing LLMs with minimal accuracy impact. Third, SwiftKV can naturally incorporate KV cache compression to improve inference performance in low-memory scenarios. Our comprehensive experiments show that SwiftKV can effectively reduce prefill computation by 25-50% across several LLM families while incurring minimum quality degradation. In the end-to-end inference serving, SwiftKV realizes up to 2x higher aggregate throughput and 60% lower time per output token. It can achieve a staggering 560 TFlops/GPU of normalized inference throughput, which translates to 16K tokens/s for Llama-3.1-70B. SwiftKV is open-sourced at https://github.com/snowflakedb/arctictraining.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 46 of its 70 references that are in the map. First posted on arXiv 4 Oct 2024.

Builds on

46 of 70 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Models23 May
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  3. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 43

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  11. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  12. 2022Locating and Editing Factual Associations in GPTcited by 2,691
31 more
  1. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  2. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  3. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  4. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  5. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  6. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  7. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  8. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  9. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  10. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  11. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  12. 2022Efficiently Scaling Transformer Inferencecited by 539
  13. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  14. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  15. 2023DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correctioncited by 477
  16. 2023Benchmarking Large Language Models for News Summarizationcited by 445
  17. 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 438
  18. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 415
  19. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  20. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
  21. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  22. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 307
  23. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decodingcited by 269
  24. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 226
  25. 2023RA-DIT: Retrieval-Augmented Dual Instruction Tuningcited by 195
  26. 2024LaCo: Large Language Model Pruning via Layer Collapsecited by 126
  27. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  28. 2022FP8 Quantization: The Power of the Exponentcited by 88
  29. 2024ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Cachingcited by 48
  30. 2024FFN-SkipLLM: A Hidden Gem for Autoregressive Decoding with Adaptive Feed Forward Skippingcited by 23
  31. 2024FoldGPT: Simple and Effective Large Language Model Compression Schemecited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 13.

19 papers.

Sort:

By area:

  1. 2026IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reusecited by 31
  2. 2026Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reusecited by 15
  3. 2025Speculative Prefill: Turbocharging TTFT with Lightweight and Training-Free Token Importance Estimationcited by 14
  4. 2025Cache Me If You Can: How Many KVs Do You Need for Effective Long-Context LMs?cited by 10
  5. 2026HySparse: A Hybrid Sparse Attention Architecture with Oracle Token Selection and KV Cache Sharingcited by 10
  6. 2026ICaRus: Identical Cache Reuse for Efficient Multi Model Inferencecited by 8
  7. 2025Shift Parallelism: Low-Latency, High-Throughput LLM Inference for Dynamic Workloadscited by 5
  8. 2026Mix-Quant: Quantized Prefilling, Precise Decoding for Agentic LLMscited by 4
  9. 2026Adaptive Layer Selection for Layer-Wise Token Pruning in LLM Inferencecited by 3
  10. 2025Arctic Inference with Shift Parallelism: Fast and Efficient Open Source Inference System for Enterprise AIcited by 2
  11. 2025P/D-Device: Disaggregated Large Language Model between Cloud and Devicescited by 2
  12. 2026Fast Forward: Accelerating LLM Prefill with Predictive FFN Sparsitycited by 1
7 more
  1. 2026KV Cache Offloading for Context-Intensive Taskscited by 1
  2. 2026Shallow Prefill, Deep Decoding: Efficient Long-Context Inference via Layer-Asymmetric KV Visibilitycited by 1
  3. 2026DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compressioncited by 1
  4. 2025HA-RAG: Hotness-Aware RAG Acceleration via Mixed Precision and Data Placementcited by 0
  5. 2026EPAS: Efficient Training with Progressive Activation Sharingcited by 0
  6. 2026FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inferencecited by 0
  7. 2026Towards Looped Models Done Right, Part II: Rethinking at Fixed Pointscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MiniKV: Pushing the Limits of LLM Inference via 2-Bit Layer-Discriminative KV Cache
  2. 2025PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applications
  3. 2025VecInfer: Efficient LLM Inference with Low-Bit KV Cache via Outlier-Suppressed Vector Quantization
  4. 2024KVPR: Efficient LLM Inference with I/O-Aware KV Cache Partial Recomputation
  5. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  6. 2025HAMburger: Accelerating LLM Inference via Token Smashing
  7. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills
  8. 2025WindowKV: Task-Adaptive Group-Wise KV Cache Window Selection for Efficient LLM Inference