mlsanityData to 14 Oct 2026

InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management

arXiv 2406.19707 ↗First posted 28 Jun 2024cs.LG cs.DCcited by 245

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Transformer-based large language models (LLMs) demonstrate impressive performance across various natural language processing tasks. Serving LLM inference for generating long contents, however, poses a challenge due to the enormous memory footprint of the transient state, known as the key-value (KV) cache, which scales with the sequence length and batch size. In this paper, we present InfiniGen, a novel KV cache management framework tailored for long-text generation, which synergistically works with modern offloading-based inference systems. InfiniGen leverages the key insight that a few important tokens that are essential for computing the subsequent attention layer in the Transformer can be speculated by performing a minimal rehearsal with the inputs of the current layer and part of the query weight and key cache of the subsequent layer. This allows us to prefetch only the essential KV cache entries (without fetching them all), thereby mitigating the fetch overhead from the host memory in offloading-based LLM serving systems. Our evaluation on several representative LLMs shows that InfiniGen improves the overall performance of a modern offloading-based system by up to 3.00x compared to prior KV cache management methods while offering substantially better model accuracy.

Key facts. Cited by 245 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 43 of its 74 references that are in the map. First posted on arXiv 28 Jun 2024.

Builds on

43 of 74 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  2. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 41

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  8. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  9. 2016Pointer Sentinel Mixture Modelscited by 3,096
  10. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  12. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
29 more
  1. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  2. 2020Rethinking Attention with Performerscited by 1,601
  3. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  4. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  5. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  6. 2018Hierarchical Neural Story Generationcited by 1,193
  7. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  8. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  9. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 828
  10. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  11. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  12. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  13. 2022Efficiently Scaling Transformer Inferencecited by 539
  14. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  15. 2016Clipper: A Low-Latency Online Prediction Serving Systemcited by 191
  16. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 181
  17. 2022A Fast Post-Training Pruning Framework for Transformerscited by 162
  18. 2018Training Millions of Personalized Dialogue Agentscited by 126
  19. 2020Serving DNNs like Clockwork: Performance Predictability from the Bottom Upcited by 120
  20. 2023OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantizationcited by 92
  21. 2023A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Modelscited by 89
  22. 2017TensorFlow-Serving: Flexible, High-Performance ML Servingcited by 85
  23. 2023Element-aware Summarization with Large Language Models: Expert-aligned Evaluation and Chain-of-Thought Methodcited by 76
  24. 2020TurboTransformers: An Efficient GPU Serving System For Transformer Modelscited by 71
  25. 2019Pretraining-Based Natural Language Generation for Text Summarizationcited by 40
  26. 2021FLAT: An Optimized Dataflow for Mitigating Attention Bottleneckscited by 39
  27. 2024Tender: Accelerating Large Language Models via Tensor Decomposition and Runtime Requantizationcited by 28
  28. 2023G10: Enabling An Efficient Unified GPU Memory and Storage Architecture with Smart Tensor Migrationscited by 10
  29. 2018Dynamic Control Flow in Large-Scale Machine Learningcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 25; 2025: 103; 2026: 117.

245 papers.

Sort:

By area:

  1. 2025Agentic Context Engineering: Evolving Contexts for Self-Improving Language Modelscited by 398
  2. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 134
  3. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
  4. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 126
  5. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 116
  6. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 97
  7. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstructioncited by 95
  8. 2024ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compressioncited by 65
  9. 2024LLM Inference Serving: Survey of Recent Advances and Opportunitiescited by 46
  10. 2024VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Accelerationcited by 43
  11. 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compressioncited by 41
  12. 2025KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Providercited by 41
38 more
  1. 2024ProMoE: Fast MoE-based LLM Serving using Proactive Cachingcited by 40
  2. 2025KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inferencecited by 40
  3. 2024TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selectioncited by 39
  4. 2025The Cost of Dynamic Reasoning: Demystifying AI Agents and Test-Time Scaling from an AI Infrastructure Perspectivecited by 39
  5. 2025Position: Episodic Memory is the Missing Piece for Long-Term LLM Agentscited by 36
  6. 2025Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generationcited by 36
  7. 2025StreamAgent: Towards Anticipatory Agents for Streaming Video Understandingcited by 33
  8. 2024Marconi: Prefix Caching for the Era of Hybrid LLMscited by 31
  9. 2025Tensor Product Attention Is All You Needcited by 31
  10. 2024MatryoshkaKV: Adaptive KV Compression via Trainable Orthogonal Projectioncited by 29
  11. 2025AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunitiescited by 28
  12. 2024DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compactioncited by 26
  13. 2025Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloadingcited by 24
  14. 2024SCOPE: Optimizing Key-Value Cache Compression in Long-context Generationcited by 20
  15. 2025LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inferencecited by 20
  16. 2024InstInfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inferencecited by 19
  17. 2025KVComm: Enabling Efficient LLM Communication through Selective KV Sharingcited by 18
  18. 2025JITServe: SLO-aware LLM Serving with Imprecise Request Informationcited by 17
  19. 2025Compactor: Calibrated Query-Agnostic KV Cache Compression with Approximate Leverage Scorescited by 17
  20. 2024LayerKV: Optimizing Large Language Model Serving with Layer-wise KV Cache Managementcited by 16
  21. 2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
  22. 2025TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applicationscited by 16
  23. 2024Locret: Enhancing Eviction in Long-Context LLM Inference with Trained Retaining Heads on Consumer-Grade Devicescited by 15
  24. 2025HydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Cloudscited by 14
  25. 2025Dialogue Injection Attack: Jailbreaking LLMs through Context Manipulationcited by 14
  26. 2025FreeKV: Boosting KV Cache Retrieval for Efficient LLM Inferencecited by 14
  27. 2025PDTrim: Targeted Pruning for Prefill-Decode Disaggregation in Inferencecited by 14
  28. 2025Pie: A Programmable Serving System for Emerging LLM Applicationscited by 14
  29. 2026Learning to Evict from Key-Value Cachecited by 14
  30. 2024KVPR: Efficient LLM Inference with I/O-Aware KV Cache Partial Recomputationcited by 13
  31. 2024PrefixKV: Adaptive Prefix KV Cache is What Vision Instruction-Following Models Need for Efficient Generationcited by 13
  32. 2025Fate: Fast Edge Inference of Mixture-of-Experts Models via Cross-Layer Gatecited by 13
  33. 2025RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inferencecited by 13
  34. 2025SlimInfer: Accelerating Long-Context LLM Inference via Dynamic Token Pruningcited by 13
  35. 2026Evaluating Memory Structure in LLM Agentscited by 12
  36. 2024TPI-LLM: Serving 70B-scale LLMs Efficiently on Low-resource Edge Devicescited by 11
  37. 2025EdgeLoRA: An Efficient Multi-Tenant LLM Serving System on Edge Devicescited by 11
  38. 2025Beluga: A CXL-Based Memory Architecture for Scalable and Efficient LLM KVCache Managementcited by 11

50 most cited of 245 All 245 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Models
  2. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
  3. 2024CORM: Cache Optimization with Recent Message for Large Language Model Inference
  4. 2024Layer-Condensed KV Cache for Efficient Inference of Large Language Models
  5. 2024A Method for Building Large Language Models with Predefined KV Cache Capacity
  6. 2024D2O: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Models
  7. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Models
  8. 2024XKV: Personalized KV Cache Memory Reduction for Long-Context LLM Inference