mlsanityData to 7 Oct 2026

MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention

arXiv 2407.02490 ↗First posted 2 Jul 2024cs.CL cs.LGcited by 375

Not on the map. Why?

Cited by 142 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The computational challenges of Large Language Model (LLM) inference remain a significant barrier to their widespread deployment, especially as prompt lengths continue to increase. Due to the quadratic complexity of the attention computation, it takes 30 minutes for an 8B LLM to process a prompt of 1M tokens (i.e., the pre-filling stage) on a single A100 GPU. Existing methods for speeding up prefilling often fail to maintain acceptable accuracy or efficiency when applied to long-context LLMs. To address this gap, we introduce MInference (Milliontokens Inference), a sparse calculation method designed to accelerate pre-filling of long-sequence processing. Specifically, we identify three unique patterns in long-context attention matrices-the A-shape, Vertical-Slash, and Block-Sparsethat can be leveraged for efficient sparse computation on GPUs. We determine the optimal pattern for each attention head offline and dynamically build sparse indices based on the assigned pattern during inference. With the pattern and sparse indices, we perform efficient sparse attention calculations via our optimized GPU kernels to significantly reduce the latency in the pre-filling stage of long-context LLMs. Our proposed technique can be directly applied to existing LLMs without any modifications to the pre-training setup or additional fine-tuning. By evaluating on a wide range of downstream tasks, including InfiniteBench, RULER, PG-19, and Needle In A Haystack, and models including LLaMA-3-1M, GLM4-1M, Yi-200K, Phi-3-128K, and Qwen2-128K, we demonstrate that MInference effectively reduces inference latency by up to 10x for pre-filling on an A100, while maintaining accuracy. Our code is available at https://aka.ms/MInference.

Key facts. Cited by 375 papers in the map (data through 7 Oct 2026). Builds on 77 of its 99 references that are in the map. First posted on arXiv 2 Jul 2024.

Builds on

77 of 99 references.

References · 77

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  4. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  5. 2020Longformer: The Long-Document Transformercited by 3,362
  6. 2023Qwen Technical Reportcited by 3,273
  7. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  8. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  9. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  10. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  11. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  12. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
38 more
  1. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  2. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  3. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  4. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  5. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  6. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  7. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  8. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  9. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  10. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  11. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  12. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  13. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  14. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  15. 2024Yi: Open Foundation Models by 01.AIcited by 706
  16. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  17. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  18. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  19. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  20. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  21. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 487
  22. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  23. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  24. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  25. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
  26. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  27. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  28. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  29. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compressioncited by 343
  30. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 302
  31. 2022UL2: Unifying Language Learning Paradigmscited by 292
  32. 2024Long-context LLMs Struggle with Long In-context Learningcited by 285
  33. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 279
  34. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  35. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 239
  36. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 228
  37. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  38. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 201

50 most cited of 77 All 77 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 34; 2025: 162; 2026: 179.

375 papers.

Sort:

By area:

  1. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  2. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 283
  3. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 230
  4. 2025Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsitycited by 204
  5. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 188
  6. 2024SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Accelerationcited by 158
  7. 2025Qwen2.5-1M Technical Reportcited by 158
  8. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 132
  9. 2025XAttention: Block Sparse Attention with Antidiagonal Scoringcited by 131
  10. 2024SeerAttention: Learning Intrinsic Sparse Attention in Your LLMscited by 121
  11. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 121
  12. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 114
38 more
  1. 2024SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantizationcited by 113
  2. 2025SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inferencecited by 113
  3. 2025VSA: Faster Video Diffusion with Trainable Sparse Attentioncited by 100
  4. 2025Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutationcited by 100
  5. 2025Agent Lightning: Train ANY AI Agents with Reinforcement Learningcited by 94
  6. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstructioncited by 89
  7. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distributioncited by 70
  8. 2025Training-free and Adaptive Sparse Attention for Efficient Long Video Generationcited by 65
  9. 2025Radial Attention: O(nlog n) Sparse Attention with Energy Decay for Long Video Generationcited by 64
  10. 2025SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Trainingcited by 63
  11. 2024Why Does the Effective Context Length of LLMs Fall Short?cited by 60
  12. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methodscited by 59
  13. 2024Squeezed Attention: Accelerating Long Context Length LLM Inferencecited by 56
  14. 2025SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse-Linear Attentioncited by 55
  15. 2024ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routingcited by 53
  16. 2024FrameFusion: Combining Similarity and Importance for Video Token Reduction on Large Vision Language Modelscited by 52
  17. 2025MiniCPM4: Ultra-Efficient LLMs on End Devicescited by 51
  18. 2025LiveVLM: Efficient Online Video Understanding via Streaming-Oriented KV Cache and Retrievalcited by 50
  19. 2024Post-Training Sparse Attention with Double Sparsitycited by 45
  20. 2024ZipVL: Efficient Large Vision-Language Models with Dynamic Token Sparsificationcited by 43
  21. 2026Xiaomi-Robotics-0: An Open-Sourced Vision-Language-Action Model with Real-Time Executioncited by 42
  22. 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compressioncited by 41
  23. 2025LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attentioncited by 41
  24. 2024Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reductioncited by 39
  25. 2024TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selectioncited by 38
  26. 2025Video-XL-2: Towards Very Long-Video Understanding Through Task-Aware KV Sparsificationcited by 36
  27. 2024LongReward: Improving Long-context Large Language Models with AI Feedbackcited by 34
  28. 2025Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate Video Diffusion Transformerscited by 34
  29. 2026MiniMax Sparse Attentioncited by 34
  30. 2025Training-Free Efficient Video Generation via Dynamic Token Carvingcited by 33
  31. 2025InfLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long Adaptationcited by 31
  32. 2025The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMscited by 28
  33. 2025Inference-Time Hyper-Scaling with KV Cache Compressioncited by 28
  34. 2025SeerAttention-R: Sparse Attention Adaptation for Long Reasoningcited by 28
  35. 2026IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reusecited by 27
  36. 2025QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cachecited by 26
  37. 2025MMInference: Accelerating Pre-filling for Long-Context VLMs via Modality-Aware Permutation Sparse Attentioncited by 26
  38. 2024Context Parallelism for Scalable Million-Token Inferencecited by 25

50 most cited of 375 All 375 (JSON)