Data to 7 Oct 2026

Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache

arXiv 2401.02669 ↗First posted 5 Jan 2024cs.DC cs.ARcited by 71

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Large Language Models (LLMs) demonstrate substantial potential across a diverse array of domains via request serving. However, as trends continue to push for expanding context sizes, the autoregressive nature of LLMs results in highly dynamic behavior of the attention layers, showcasing significant differences in computational characteristics and memory requirements from the non-attention layers. This presents substantial challenges for resource management and performance optimization in service systems. Existing static model parallelism and resource allocation strategies fall short when dealing with this dynamicity. To address the issue, we propose Infinite-LLM, a novel LLM serving system designed to effectively handle dynamic context lengths. Infinite-LLM disaggregates attention layers from an LLM's inference process, facilitating flexible and independent resource scheduling that optimizes computational performance and enhances memory utilization jointly. By leveraging a pooled GPU memory strategy across a cluster, Infinite-LLM not only significantly boosts system throughput but also supports extensive context lengths. Evaluated on a dataset with context lengths ranging from a few to 2000K tokens across a cluster with 32 A100 GPUs, Infinite-LLM demonstrates throughput improvement of 1.35-3.4x compared to state-of-the-art methods, enabling efficient and elastic LLM deployment.

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 34 of its 67 references that are in the map. First posted on arXiv 5 Jan 2024.

Builds on

34 of 67 references.

Closest ancestors · 1

Same direction, newest first

2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills31 Aug

Other references · 33

2017Attention Is All You Needcited by 59,932
2020Language Models are Few-Shot Learnerscited by 32,199
2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
2021Evaluating Large Language Models Trained on Codecited by 8,053
2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
2020Longformer: The Long-Document Transformercited by 3,362
2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
2019Generating Long Sequences with Sparse Transformerscited by 1,523
2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378

21 more

2023PaLM 2 Technical Reportcited by 1,154
2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
2022Efficiently Scaling Transformer Inferencecited by 529
2017Ray: A Distributed Framework for Emerging AI Applicationscited by 516
2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
2023Benchmarking Large Language Models for News Summarizationcited by 444
2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
2023Large Language Models for Information Retrieval: A Surveycited by 331
2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 177
2020Memory-Efficient Pipeline-Parallel DNN Trainingcited by 133
2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
2021Sequence Parallelism: Long Sequence Training from System Perspectivecited by 125
2020DAPPLE: A Pipelined Data Parallel Approach for Training Large Modelscited by 100
2023Element-aware Summarization with Large Language Models: Expert-aligned Evaluation and Chain-of-Thought Methodcited by 76
2023GPTEval: A Survey on Assessments of ChatGPT and GPT-4cited by 73
2023Striped Attention: Faster Ring Attention for Causal Transformerscited by 52
2023Leveraging Large Language Models to Power Chatbots for Collecting User Self-Reported Datacited by 48
2013Petuum: A New Platform for Distributed Machine Learning on Big Datacited by 35
2023Beyond the Hype: Assessing the Performance, Trustworthiness, and Clinical Suitability of GPT3.5cited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 28; 2025: 25; 2026: 18.

71 papers.

Sort:

By area:

2024A Survey on Efficient Inference for Large Language Modelscited by 197
2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
2025A Survey of Context Engineering for Large Language Modelscited by 120
2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 96
2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compressioncited by 84
2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Poolcited by 67
2024Why Does the Effective Context Length of LLMs Fall Short?cited by 60
2024FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelinescited by 49
2024LLM Inference Serving: Survey of Recent Advances and Opportunitiescited by 46
2024Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Servicescited by 44
2024Challenges in Deploying Long-Context Transformers: A Theoretical Peak Performance Analysiscited by 44
2025Position: Episodic Memory is the Missing Piece for Long-Term LLM Agentscited by 35

38 more

2024Teola: Towards End-to-End Optimization of LLM-based Applicationscited by 32
2024NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokenscited by 31
2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
2025MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelismcited by 31
2024DAPE: Data-Adaptive Positional Encoding for Length Extrapolationcited by 30
2025Cognitive Memory in Large Language Modelscited by 28
2024Wings: Learning Multimodal LLMs without Text-only Forgettingcited by 23
2024InstInfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inferencecited by 19
2025Shadow in the Cache: Unveiling and Mitigating Privacy Risks of KV-cache in LLM Inferencecited by 19
2024Stylus: Automatic Adapter Selection for Diffusion Modelscited by 18
2024Lexico: Extreme KV Cache Compression via Sparse Coding over Universal Dictionariescited by 17
2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
2025Seesaw: High-throughput LLM Inference via Model Re-shardingcited by 15
2024vTensor: Flexible Virtual Tensor Management for Efficient LLM Servingcited by 13
2024Pie: Pooling CPU Memory for LLM Inferencecited by 13
2024Adaptive Layer Splitting for Wireless LLM Inference in Edge Computing: A Model-Based Reinforcement Learning Approachcited by 12
2025KV Cache Compression for Inference Efficiency in LLMs: A Reviewcited by 12
2024DAPE V2: Process Attention Score as Feature Map for Length Extrapolationcited by 10
2025HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-locationcited by 10
2024LLMSteer: Improving Long-Context LLM Inference by Steering Attention on Reused Contextscited by 7
2025Lizard: An Efficient Linearization Framework for Large Language Modelscited by 6
2025PICE: A Semantic-Driven Progressive Inference System for LLM Serving in Cloud-Edge Networkscited by 5
2025TokenLake: A Unified Segment-level Prefix Cache Pool for Fine-grained Elastic Long-Context LLM Servingcited by 5
2024QuickLLaMA: Query-aware Inference Acceleration for Large Language Modelscited by 4
2025LLMQuoter: Enhancing RAG Capabilities Through Efficient Quote Extraction From Large Contextscited by 4
2025From Standalone LLMs to Integrated Intelligence: A Survey of Compound Al Systemscited by 4
2025MCBP: A Memory-Compute Efficient LLM Inference Accelerator Leveraging Bit-Slice-enabled Sparsity and Repetitivenesscited by 4
2026Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Servingcited by 4
2024The CAP Principle for LLM Serving: A Survey of Long-Context Large Language Model Servingcited by 3
2025MegaScale-Data: Scaling Dataloader for Multisource Large Foundation Model Trainingcited by 3
2024Inference Acceleration for Large Language Models on CPUscited by 2
2024XKV: Personalized KV Cache Memory Reduction for Long-Context LLM Inferencecited by 2
2025DiSCo: Device-Server Collaborative LLM-Based Text Streaming Servicescited by 2
2025SpecRouter: Adaptive Routing for Multi-Level Speculative Decoding in Large Language Modelscited by 2
2025KernelBand: Steering LLM-based Kernel Optimization via Hardware-Aware Multi-Armed Banditscited by 2
2025WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Servingcited by 2
2026Agent-Assisted Side-Channel Attacks on Non-Prefix KV Cache in RAGcited by 2
2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimizationcited by 2

50 most cited of 71 All 71 (JSON)

Similar, not cited

Close in wording, no citation either way