mlsanityData to 14 Oct 2026

HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloading

arXiv 2502.12574 ↗First posted 18 Feb 2025cs.LG cs.AIcited by 9

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Transformer-based large language models (LLMs) demonstrate impressive performance in long context generation. Extending the context length has disproportionately shifted the memory footprint of LLMs during inference to the key-value cache (KV cache). In this paper, we propose HEADINFER, which offloads the KV cache to CPU RAM while avoiding the need to fully store the KV cache for any transformer layer on the GPU. HEADINFER employs a fine-grained, head-wise offloading strategy, maintaining only selective attention heads KV cache on the GPU while computing attention output dynamically. Through roofline analysis, we demonstrate that HEADINFER maintains computational efficiency while significantly reducing memory footprint. We evaluate HEADINFER on the Llama-3-8B model with a 1-million-token sequence, reducing the GPU memory footprint of the KV cache from 128 GB to 1 GB and the total GPU memory usage from 207 GB to 17 GB, achieving a 92% reduction compared to BF16 baseline inference. Notably, HEADINFER enables 4-million-token inference with an 8B model on a single consumer GPU with 24GB memory (e.g., NVIDIA RTX 4090) without approximation methods.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 43 of its 52 references that are in the map. First posted on arXiv 18 Feb 2025.

Builds on

43 of 52 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methods13 Dec
  2. 2024ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression4 Dec
  3. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference28 Oct
  4. 2024Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning25 Oct
  5. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
  6. 2024LayerKV: Optimizing Large Language Model Serving with Layer-wise KV Cache Management1 Oct
  7. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Heads22 Jul
  8. 2024vTensor: Flexible Virtual Tensor Management for Efficient LLM Serving22 Jul
  9. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  10. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  6. 2020Longformer: The Long-Document Transformercited by 3,413
  7. 2023Mistral 7Bcited by 3,164
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  9. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  10. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  11. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  12. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
21 more
  1. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  2. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  3. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  4. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  5. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  6. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  7. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  8. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  9. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  10. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  11. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  12. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 242
  13. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 232
  14. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  15. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 204
  16. 2024World Model on Million-Length Video And Language With Blockwise RingAttentioncited by 198
  17. 2023PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Trainingcited by 111
  18. 2024FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelinescited by 49
  19. 2024NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inferencecited by 37
  20. 2023Giraffe: Adventures in Expanding Context Lengths in LLMscited by 33
  21. 2024LoQT: Low-Rank Adapters for Quantized Pretrainingcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 5.

9 papers.

  1. 2026A Survey on the Green Development of Large Models: From Resource-Efficient Architectures to Hardware-Software Co-Designcited by 5
  2. 2025HGCA: Hybrid GPU-CPU Attention for Long Context LLM Inferencecited by 4
  3. 2025Adaptive and Resource-efficient Agentic AI Systems for Mobile and Embedded Devices: A Surveycited by 2
  4. 2025Federated Attention: A Distributed Paradigm for Collaborative LLM Inference over Edge Networkscited by 1
  5. 2026Not All Thoughts Need HBM: Semantics-Aware Memory Hierarchy for LLM Reasoningcited by 1
  6. 2025MOM: Memory-Efficient Offloaded Mini-Sequence Inference for Long Context Language Modelscited by 0
  7. 2026MCAP: Deployment-Time Layer Profiling for Memory-Constrained LLM Inferencecited by 0
  8. 2026Physically Partitioned KVCache Format for CPU--GPU Load Balancing in MoE Inferencecited by 0
  9. 2026WakeKV: Reactive, Reversible KV Residency for Heads That Change Their Mindscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  2. 2024XKV: Personalized KV Cache Memory Reduction for Long-Context LLM Inference
  3. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving
  4. 2025LLaMCAT: Optimizing Large Language Model Inference with Cache Arbitration and Throttling
  5. 2025BaKlaVa -- Budgeted Allocation of KV cache for Long-context Inference
  6. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference
  7. 2026Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Serving
  8. 2026PackInfer: Compute- and I/O-Efficient Attention for Batched LLM Inference