mlsanityData to 14 Oct 2026

Star Attention: Efficient LLM Inference over Long Sequences

arXiv 2411.17116 ↗First posted 26 Nov 2024cs.CL cs.AI cs.LGcited by 25

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Inference with Transformer-based Large Language Models (LLMs) on long sequences is both costly and slow due to the quadratic complexity of the self-attention mechanism. We introduce Star Attention, a two-phase block-sparse approximation that improves computational efficiency by sharding attention across multiple hosts while minimizing communication overhead. In the first phase, the context is processed using blockwise-local attention across hosts, in parallel. In the second phase, query and response tokens attend to all prior cached tokens through sequence-global attention. Star Attention integrates seamlessly with most Transformer-based LLMs trained with global attention, reducing memory requirements and inference time by up to 11x while preserving 97-100% of accuracy.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 28 of its 41 references that are in the map. First posted on arXiv 26 Nov 2024.

Builds on

28 of 41 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  2. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  3. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 25

  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  2. 2020Longformer: The Long-Document Transformercited by 3,413
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  8. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  9. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  10. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  11. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  12. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
13 more
  1. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  2. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 215
  3. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 200
  4. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 197
  5. 2018Online normalizer calculation for softmaxcited by 162
  6. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 144
  7. 2021Sequence Parallelism: Long Sequence Training from System Perspectivecited by 126
  8. 2023LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Modelscited by 124
  9. 2024Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Modelscited by 57
  10. 2024ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Cachingcited by 48
  11. 2024Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusterscited by 13
  12. 2024E2LLM: Encoder Elongated Large Language Models for Long-Context Understanding and Reasoningcited by 13
  13. 2024Writing in the Margins: Better Inference Pattern for Long Context Retrievalcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 14; 2026: 10.

25 papers.

Sort:

By area:

  1. 2025A Survey of Context Engineering for Large Language Modelscited by 126
  2. 2025Training-free and Adaptive Sparse Attention for Efficient Long Video Generationcited by 69
  3. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methodscited by 60
  4. 2025MMInference: Accelerating Pre-filling for Long-Context VLMs via Modality-Aware Permutation Sparse Attentioncited by 26
  5. 2025ParallelComp: Parallel Long-Context Compressor for Length Extrapolationcited by 14
  6. 2025Efficient Many-Shot In-Context Learning with Dynamic Block-Sparse Attentioncited by 12
  7. 2025FlashSVD: Memory-Efficient Inference with Streaming for Low-Rank Modelscited by 9
  8. 2025Attention Mechanisms Perspective: Exploring LLM Processing of Graph-Structured Datacited by 8
  9. 2025Focusing by Contrastive Attention: Enhancing VLMs' Visual Reasoningcited by 8
  10. 2025Delta Attention: Fast and Accurate Sparse Attention Inference by Delta Correctioncited by 7
  11. 2025FSA: An Alternative Efficient Implementation of Native Sparse Attention Kernelcited by 7
  12. 2025Attention Is Not All You Need: The Importance of Feedforward Networks in Transformer Modelscited by 4
13 more
  1. 2025CTR-Sink: Attention Sink for Language Models in Click-Through Rate Predictioncited by 4
  2. 2025APB: Accelerating Distributed Long-Context Inference by Passing Compressed Context Blocks across GPUscited by 3
  3. 2025Scaling Down, Serving Fast: Compressing and Deploying Efficient LLMs for Recommendation Systemscited by 2
  4. 2026LazyAttention: Efficient Retrieval-Augmented Generation with Deferred Positional Encodingcited by 2
  5. 2026More Than a Quick Glance: Overcoming the Greedy Bias in KV-Cache Compressioncited by 1
  6. 2026Benchmarking Compact VLMs for Clip-Level Surveillance Anomaly Detection Under Weak Supervisioncited by 1
  7. 2026AdapShot: Adaptive Many-Shot In-Context Learning with Semantic-Aware KV Cache Reusecited by 1
  8. 2026Overlapping Schwarz Attention: Hierarchical Attention via Domain Decompositioncited by 1
  9. 2026APB-V: Accelerating Long-Video Understanding via Sequence-Parallelism-aware Approximate Attentioncited by 0
  10. 2026Do Vision Transformers Need All-to-All Attention? Global Communication Through Elastic Learned Corescited by 0
  11. 2026Dropping the Anchor: Statistical Context Summarization for Distributed Systems via Pulsar Attentioncited by 0
  12. 2026No More Free Lunch: Corpus Task Complexity Matters as Corpora Growcited by 0
  13. 2026HHR: Hierarchical Hash Retrieval for Efficient LLM Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023SparQ Attention: Bandwidth-Efficient LLM Inference
  2. 2024Beyond KV Caching: Shared Attention for Efficient LLMs
  3. 2025LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention
  4. 2024A Survey on Efficient Inference for Large Language Models
  5. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline
  6. 2025MoSKA: Mixture of Shared KV Attention for Efficient Long-Sequence LLM Inference
  7. 2025SlimInfer: Accelerating Long-Context LLM Inference via Dynamic Token Pruning
  8. 2025SALE : Low-bit Estimation for Efficient Sparse Attention in Long-context LLM Prefilling