mlsanityData to 7 Oct 2026

InstInfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inference

arXiv 2409.04992 ↗First posted 8 Sep 2024cs.AR cs.CLcited by 19

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

The widespread of Large Language Models (LLMs) marks a significant milestone in generative AI. Nevertheless, the increasing context length and batch size in offline LLM inference escalate the memory requirement of the key-value (KV) cache, which imposes a huge burden on the GPU VRAM, especially for resource-constraint scenarios (e.g., edge computing and personal devices). Several cost-effective solutions leverage host memory or SSDs to reduce storage costs for offline inference scenarios and improve the throughput. Nevertheless, they suffer from significant performance penalties imposed by intensive KV cache accesses due to limited PCIe bandwidth. To address these issues, we propose InstInfer, a novel LLM inference system that offloads the most performance-critical computation (i.e., attention in decoding phase) and data (i.e., KV cache) parts to Computational Storage Drives (CSDs), which minimize the enormous KV transfer overheads. InstInfer designs a dedicated flash-aware in-storage attention engine with KV cache management mechanisms to exploit the high internal bandwidths of CSDs instead of being limited by the PCIe bandwidth. The optimized P2P transmission between GPU and CSDs further reduces data migration overheads. Experimental results demonstrate that for a 13B model using an NVIDIA A6000 GPU, InstInfer improves throughput for long-sequence inference by up to 11.1×, compared to existing SSD-based solutions such as FlexGen.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 37 of its 75 references that are in the map. First posted on arXiv 8 Sep 2024.

Builds on

37 of 75 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Heads22 Jul
  2. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  3. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention23 Mar
  4. 2023SparQ Attention: Bandwidth-Efficient LLM Inference8 Dec
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  6. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun
  7. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  6. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  7. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  8. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  9. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  10. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  11. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  12. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
18 more
  1. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  2. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  3. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  4. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 170
  5. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  6. 2024LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuningcited by 133
  7. 2019An Attentive Survey of Attention Modelscited by 123
  8. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  9. 2024ServerlessLLM: Low-Latency Serverless Inference for Large Language Modelscited by 78
  10. 2023Element-aware Summarization with Large Language Models: Expert-aligned Evaluation and Chain-of-Thought Methodcited by 76
  11. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCachecited by 71
  12. 2024Hydragen: High-Throughput LLM Inference with Shared Prefixescited by 71
  13. 2024FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelinescited by 49
  14. 2024NeuPIMs: NPU-PIM Heterogeneous Acceleration for Batched LLM Inferencingcited by 48
  15. 2024LLM-PQ: Serving LLM on Heterogeneous Clusters with Phase-Aware Partition and Adaptive Quantizationcited by 14
  16. 2022SmartSAGE: Training Large-scale Graph Neural Networks using In-Storage Processing Architecturescited by 12
  17. 2023PIM-GPT: A Hybrid Process-in-Memory Accelerator for Autoregressive Transformerscited by 8
  18. 2020ZnG: Architecting GPU Multi-Processors with New Flash for Scalable Data Analysiscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 8; 2026: 9.

19 papers.

Sort:

By area:

  1. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  2. 2025Cognitive Memory in Large Language Modelscited by 28
  3. 2025PAPI: Exploiting Dynamic Parallelism in Large Language Model Decoding with a Processing-In-Memory-Enabled Computing Systemcited by 19
  4. 2025LittleBit: Ultra Low-Bit Quantization via Latent Factorizationcited by 11
  5. 2025EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Servingcited by 9
  6. 2026PackCache: A Training-Free Acceleration Method for Unified Autoregressive Video Generation via Compact KV-Cachecited by 9
  7. 2024Harnessing Your DRAM and SSD for Sustainable and Accessible LLM Inference with Mixed-Precision and Multi-level Cachingcited by 3
  8. 2025Cost-Efficient LLM Serving in the Cloud: VM Selection with KV Cache Offloadingcited by 2
  9. 2026DUAL-BLADE: Dual-Path NVMe-Direct KV-Cache Offloading for Edge LLM Inferencecited by 2
  10. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimizationcited by 2
  11. 2025iServe: An Intent-based Serving System for LLMscited by 1
  12. 2025Harmonia: Enhancing Data Placement and Migration in Hybrid Storage Systems via Multi-Agent Reinforcement Learningcited by 1
7 more
  1. 2025Symbiosis: Multi-Adapter Inference and Fine-Tuningcited by 0
  2. 2026Efficient Remote KV Cache Reuse with GPU-native Video Codeccited by 0
  3. 2026SinkRouter: Sink-Aware Routing for Efficient Long-Context Decoding in Large Language and Multimodal Modelscited by 0
  4. 2026Can I Buy Your KV Cache?cited by 0
  5. 2026Unified KV Pooling to Accelerate Long-Context LLM Servingcited by 0
  6. 2026NELSSA: A GPU-PNM Heterogeneous System for Mixed-Length LLM Serving via Length-based Request Placementcited by 0
  7. 2026Q-First: Most of Attention Needs Only the Query in Disaggregated LLM Decodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Cost-Effective Near-Storage Processing Solution for Offline Inference of Long-Context LLMs
  2. 2025HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloading
  3. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving
  4. 2026ScoutAttention: Efficient KV Cache Offloading via Layer-Ahead CPU Pre-computation for LLM Inference
  5. 2025Reimagining Memory Access for LLM Inference: Compression-Aware Memory Controller Design
  6. 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference
  7. 2026HybridGen: Efficient LLM Generative Inference via CPU-GPU Hybrid Computing
  8. 2024XKV: Personalized KV Cache Memory Reduction for Long-Context LLM Inference