# LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention

arXiv: https://arxiv.org/abs/2502.14866 · HTML: https://mlsanity.com/paper/2502.14866/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving.
- Builds on 40 of its 47 references that are in the map.
- First posted on arXiv 20 Feb 2025.

## Placed in

- Efficient LLM inference and serving / [KV cache optimization](https://mlsanity.com/inference/kv-cache-optimization/)

## Builds on (most cited first, 30 of 40)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://mlsanity.com/paper/2408.03314/) (2024)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models](https://mlsanity.com/paper/2306.14048/) (2023)
- [StarCoder: may the source be with you!](https://mlsanity.com/paper/2305.06161/) (2023)
- [VILA: On Pre-training for Visual Language Models](https://mlsanity.com/paper/2312.07533/) (2023)
- [KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache](https://mlsanity.com/paper/2402.02750/) (2024)
- [KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization](https://mlsanity.com/paper/2401.18079/) (2024)
- [StarCoder 2 and The Stack v2: The Next Generation](https://mlsanity.com/paper/2402.19173/) (2024)
- [Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference](https://mlsanity.com/paper/2406.10774/) (2024)
- [Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs](https://mlsanity.com/paper/2310.01801/) (2023)
- [Benchmarking Large Language Models for News Summarization](https://mlsanity.com/paper/2301.13848/) (2023)
- [MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention](https://mlsanity.com/paper/2407.02490/) (2024)
- [Efficient Attentions for Long Document Summarization](https://mlsanity.com/paper/2104.02112/) (2021)
- [LongVILA: Scaling Long-Context Visual Language Models for Long Videos](https://mlsanity.com/paper/2408.10188/) (2024)
- [DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads](https://mlsanity.com/paper/2410.10819/) (2024)
- [Retrieval Head Mechanistically Explains Long-Context Factuality](https://mlsanity.com/paper/2404.15574/) (2024)
- [PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU](https://mlsanity.com/paper/2312.12456/) (2023)
- [QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving](https://mlsanity.com/paper/2405.04532/) (2024)
- [Compact Language Models via Pruning and Knowledge Distillation](https://mlsanity.com/paper/2407.14679/) (2024)

## Built on by (most cited first, 30 of 41)

- [A Survey of Context Engineering for Large Language Models](https://mlsanity.com/paper/2507.13334/) (2025)
- [Radial Attention: O(nlog n) Sparse Attention with Energy Decay for Long Video Generation](https://mlsanity.com/paper/2506.19852/) (2025)
- [IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse](https://mlsanity.com/paper/2603.12201/) (2026)
- [SeerAttention-R: Sparse Attention Adaptation for Long Reasoning](https://mlsanity.com/paper/2506.08889/) (2025)
- [Bidirectional Sparse Attention for Faster Video Diffusion Training](https://mlsanity.com/paper/2509.01085/) (2025)
- [Energy Use of AI Inference, Efficiency Pathways, and Test-Time Scaling](https://mlsanity.com/paper/2509.20241/) (2025)
- [Kascade: A Practical Sparse Attention Method for Long-Context LLM Inference](https://mlsanity.com/paper/2512.16391/) (2025)
- [HySparse: A Hybrid Sparse Attention Architecture with Oracle Token Selection and KV Cache Sharing](https://mlsanity.com/paper/2602.03560/) (2026)
- [SpecPV: Improving Self-Speculative Decoding for Long-Context Generation via Partial Verification](https://mlsanity.com/paper/2512.02337/) (2025)
- [Compact Attention: Exploiting Structured Spatio-Temporal Sparsity for Fast Video Generation](https://mlsanity.com/paper/2508.12969/) (2025)
- [FlexiCache: Leveraging Temporal Stability of Attention Heads for Efficient KV Cache Management](https://mlsanity.com/paper/2511.00868/) (2025)
- [Double-P: Hierarchical Top-P Sparse Attention for Long-Context LLMs](https://mlsanity.com/paper/2602.05191/) (2026)
- [Self-Indexing KVCache: Predicting Sparse Attention from Compressed Keys](https://mlsanity.com/paper/2603.14224/) (2026)
- [VeriCache: Turning Lossy KV Cache into Lossless LLM Inference](https://mlsanity.com/paper/2605.17613/) (2026)
- [KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference](https://mlsanity.com/paper/2605.18071/) (2026)
- [PLM: Efficient Peripheral Language Models Hardware-Co-Designed for Ubiquitous Computing](https://mlsanity.com/paper/2503.12167/) (2025)
- [AnchorAttention: Difference-Aware Sparse Attention with Stripe Granularity](https://mlsanity.com/paper/2505.23520/) (2025)
- [Retrospective Sparse Attention for Efficient Long-Context Generation](https://mlsanity.com/paper/2508.09001/) (2025)
- [ShadowNPU: System and Algorithm Co-design for NPU-Centric On-Device LLM Inference](https://mlsanity.com/paper/2508.16703/) (2025)
- [NOSA: Native and Offloadable Sparse Attention](https://mlsanity.com/paper/2510.13602/) (2025)
- [KV Admission: Learning What to Write for Efficient Long-Context LLM Inference](https://mlsanity.com/paper/2512.17452/) (2025)
- [Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Serving](https://mlsanity.com/paper/2604.26837/) (2026)
- [From Static Inference to Dynamic Interaction: A Survey of Streaming Large Language Models](https://mlsanity.com/paper/2603.04592/) (2026)
- [P/D-Device: Disaggregated Large Language Model between Cloud and Devices](https://mlsanity.com/paper/2508.09035/) (2025)
- [SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking](https://mlsanity.com/paper/2609.13141/) (2026)
- [Training-free Context-adaptive Attention for Efficient Long Context Modeling](https://mlsanity.com/paper/2512.09238/) (2025)
- [Vortex: Efficient and Programmable Sparse Attention Serving for AI Agents](https://mlsanity.com/paper/2606.06453/) (2026)
- [LLM-CoOpt: A Co-Design and Optimization Framework for Efficient LLM Inference on Heterogeneous Platforms](https://mlsanity.com/paper/2602.09323/) (2026)
- [WANSpec: Leveraging Global Compute Capacity for LLM Inference](https://mlsanity.com/paper/2602.18931/) (2026)
- [Understand and Accelerate Memory Processing Pipeline for Large Language Model Inference](https://mlsanity.com/paper/2603.29002/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have shown remarkable potential in processing long sequences and complex reasoning tasks, yet efficiently serving these models remains challenging due to the quadratic computational complexity of attention in the prefilling stage and the large memory footprint of the KV cache in the decoding stage. To address these issues, we introduce LServe, an efficient system that accelerates long-sequence LLM serving via hybrid sparse attention. This method unifies different hardware-friendly, structured sparsity patterns for both prefilling and decoding attention into a single framework, where computations on less important tokens are skipped block-wise. LServe demonstrates the compatibility of static and dynamic sparsity in long-context LLM attention. This design enables multiplicative speedups by combining these optimizations. Specifically, we convert half of the attention heads to nearly free streaming heads in both the prefilling and decoding stages. Additionally, we find that only a constant number of KV pages is required to preserve long-context and reasoning capabilities, irrespective of context length. We then design a hierarchical KV page selection policy that dynamically prunes KV pages based on query-centric similarity. On average, LServe accelerates LLM prefilling by up to 2.9x and decoding by 1.3-2.1x over vLLM, maintaining long-context accuracy. Code is released at https://github.com/mit-han-lab/omniserve.