mlsanityData to 7 Oct 2026

FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inference

arXiv 2502.20766 ↗First posted 28 Feb 2025cs.LG cs.CLcited by 121

Not on the map. Why?

Cited by 26 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs) encounter computational challenges during long-sequence inference, especially in the attention pre-filling phase, where the complexity grows quadratically with the prompt length. Previous efforts to mitigate these challenges have relied on fixed sparse attention patterns or identifying sparse attention patterns based on limited cases. However, these methods lacked the flexibility to efficiently adapt to varying input demands. In this paper, we introduce FlexPrefill, a Flexible sparse Pre-filling mechanism that dynamically adjusts sparse attention patterns and computational budget in real-time to meet the specific requirements of each input and attention head. The flexibility of our method is demonstrated through two key innovations: 1) Query-Aware Sparse Pattern Determination: By measuring Jensen-Shannon divergence, this component adaptively switches between query-specific diverse attention patterns and predefined attention patterns. 2) Cumulative-Attention Based Index Selection: This component dynamically selects query-key indexes to be computed based on different attention patterns, ensuring the sum of attention scores meets a predefined threshold. FlexPrefill adaptively optimizes the sparse pattern and sparse ratio of each attention head based on the prompt, enhancing efficiency in long-sequence inference tasks. Experimental results show significant improvements in both speed and accuracy over prior methods, providing a more flexible and efficient solution for LLM inference.

Key facts. Cited by 121 papers in the map (data through 7 Oct 2026). Builds on 61 of its 69 references that are in the map. First posted on arXiv 28 Feb 2025.

Builds on

61 of 69 references.

References · 61

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  6. 2020Longformer: The Long-Document Transformercited by 3,362
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  8. 2023Mistral 7Bcited by 3,065
  9. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  10. 2023A Survey of Large Language Modelscited by 2,537
  11. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  12. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
38 more
  1. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  2. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  3. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  4. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  5. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  6. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  7. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  8. 2024Yi: Open Foundation Models by 01.AIcited by 706
  9. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  10. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  11. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  12. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  13. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 487
  14. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  15. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  16. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
  17. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  18. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  19. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  20. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compressioncited by 343
  21. 2024xLSTM: Extended Long Short-Term Memorycited by 320
  22. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 302
  23. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  24. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  25. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  26. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 201
  27. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 194
  28. 2023Augmenting Language Models with Long-Term Memorycited by 186
  29. 2023Compressing Context to Enhance Inference Efficiency of Large Language Modelscited by 185
  30. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 183
  31. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 175
  32. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 155
  33. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  34. 2023LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Modelscited by 120
  35. 2023SparQ Attention: Bandwidth-Efficient LLM Inferencecited by 119
  36. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 115
  37. 2023Retrieval meets Long Context Large Language Modelscited by 114
  38. 2023Scaling Transformer to 1M tokens and beyond with RMTcited by 97

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 44; 2026: 77.

121 papers.

Sort:

By area:

  1. 2025XAttention: Block Sparse Attention with Antidiagonal Scoringcited by 131
  2. 2025Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutationcited by 100
  3. 2025Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate Video Diffusion Transformerscited by 34
  4. 2026MiniMax Sparse Attentioncited by 34
  5. 2025InfLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long Adaptationcited by 31
  6. 2025The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMscited by 28
  7. 2025SeerAttention-R: Sparse Attention Adaptation for Long Reasoningcited by 28
  8. 2026IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reusecited by 27
  9. 2025MMInference: Accelerating Pre-filling for Long-Context VLMs via Modality-Aware Permutation Sparse Attentioncited by 26
  10. 2025Block-Sparse Global Attention for Efficient Multi-View Geometry Transformerscited by 22
  11. 2025Efficient Attention Mechanisms for Large Language Models: A Surveycited by 21
  12. 2026SLA2: Sparse-Linear Attention with Learnable Routing and QATcited by 21
38 more
  1. 2026Flow caching for autoregressive video generationcited by 19
  2. 2025Long-Context Generalization with Sparse Attentioncited by 18
  3. 2025Bidirectional Sparse Attention for Faster Video Diffusion Trainingcited by 18
  4. 2025PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and Quantized Attention in Visual Generation Modelscited by 17
  5. 2026SpargeAttention2: Trainable Sparse Attention via Hybrid Top-k+Top-p Masking and Distillation Fine-Tuningcited by 16
  6. 2025PDTrim: Targeted Pruning for Prefill-Decode Disaggregation in Inferencecited by 14
  7. 2025SlimInfer: Accelerating Long-Context LLM Inference via Dynamic Token Pruningcited by 13
  8. 2025BLASST: Dynamic BLocked Attention Sparsity via Softmax Thresholdingcited by 12
  9. 2026Geometry-Aware Rotary Position Embedding for Consistent Video World Modelcited by 12
  10. 2025Re-ttention: Ultra Sparse Visual Generation via Attention Statistical Reshapecited by 10
  11. 2025VideoNSA: Native Sparse Attention Scales Video Understandingcited by 10
  12. 2026HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attentioncited by 10
  13. 2025ProxyAttn: Guided Sparse Attention via Representative Headscited by 9
  14. 2026FlashPrefill: Instantaneous Pattern Discovery and Thresholding for Ultra-Fast Long-Context Prefillingcited by 9
  15. 2026ReST-KV: Robust KV Cache Eviction with Layer-wise Output Reconstruction and Spatial-Temporal Smoothingcited by 9
  16. 2025ZSMerge: Zero-Shot KV Cache Compression for Memory-Efficient Long-Context LLMscited by 8
  17. 2026Mixture of Distributions Matters: Dynamic Sparse Attention for Efficient Video Diffusion Transformerscited by 8
  18. 2025FSA: An Alternative Efficient Implementation of Native Sparse Attention Kernelcited by 7
  19. 2025Draft-based Approximate Inference for LLMscited by 6
  20. 2026DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generationcited by 6
  21. 2026Sol-Attn: Accelerating Video Generation Inference via On-the-Fly Attention Sparsificationcited by 6
  22. 2025LOOM-Scope: a comprehensive and efficient LOng-cOntext Model evaluation frameworkcited by 5
  23. 2025Compact Attention: Exploiting Structured Spatio-Temporal Sparsity for Fast Video Generationcited by 5
  24. 2025Revisiting Long-context Modeling from Context Denoising Perspectivecited by 5
  25. 2025EfficientNav: Towards On-Device Object-Goal Navigation with Navigation Map Caching and Retrievalcited by 5
  26. 2025Sparser Block-Sparse Attention via Token Permutationcited by 5
  27. 2026AngelSlim: A more accessible, comprehensive, and efficient toolkit for large model compressioncited by 5
  28. 2026Stem: Rethinking Causal Information Flow in Sparse Attentioncited by 5
  29. 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inferencecited by 5
  30. 2025DeltaLLM: A Training-Free Framework Exploiting Temporal Sparsity for Efficient Edge LLM Inferencecited by 4
  31. 2025Accelerating Prefilling via Decoding-time Contribution Sparsitycited by 4
  32. 2025ShadowNPU: System and Algorithm Co-design for NPU-Centric On-Device LLM Inferencecited by 4
  33. 2025Adamas: Hadamard Sparse Attention for Efficient Long-Context Inferencecited by 4
  34. 2026Elastic Attention: Test-time Adaptive Sparsity Ratios for Efficient Transformerscited by 4
  35. 2026Scout Before You Attend: Sketch-and-Walk Sparse Attention for Efficient LLM Inferencecited by 4
  36. 2026Switch Attention: Towards Dynamic and Fine-grained Hybrid Transformerscited by 4
  37. 2026SpotAttention: Plug-In Block-Sparse Routing for Pretrained Long-Context Transformerscited by 4
  38. 2025SALE : Low-bit Estimation for Efficient Sparse Attention in Long-context LLM Prefillingcited by 3

50 most cited of 121 All 121 (JSON)