mlsanityData to 14 Oct 2026

FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM Serving

arXiv 2608.19758 ↗First posted 20 Aug 2026cs.CLcited by 2

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Long-context modeling is a pivotal capability for Large Language Models, yet the quadratic complexity of attention remains a critical bottleneck, particularly during the compute-intensive prefilling phase. Our previous work, FlashPrefill, mitigates this cost through instantaneous pattern discovery and max-based dynamic thresholding; however, it remains an algorithmic prototype that is still distant from production deployment. In this paper, we present FlashPrefill V2, which evolves FlashPrefill from a prototype toward practical long-context serving along three dimensions. First, we introduce a mean correction term that effectively suppresses the approximation error, keeping performance degradation manageable even at extreme sparsity levels. Second, we redesign the sparse attention operator with PackGQA memory access, warp specialization, and pingpong pipelining, fully aligning with the latest FlashAttention-3/4 implementations and supporting FP8 inference to meet practical quantization requirements. Third, FlashPrefill V2 natively supports paged KV cache and continuous batching, allowing integration as an attention backend in modern inference frameworks such as SGLang. Extensive evaluations on NVIDIA H20 GPUs---among the most widely deployed inference accelerators---demonstrate that FlashPrefill V2 delivers up to 47.26x and 27.19x speedups over FlashAttention-2 at 128K context length under FP8 and BF16 precision, respectively, and, in FP8, still achieves a 30.49x speedup against an FA3/4-aligned dense baseline.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 42 of its 44 references that are in the map. First posted on arXiv 20 Aug 2026.

Builds on

42 of 44 references.

Closest ancestors · 9

Same direction, newest first

  1. 2026UniPrefill: Universal Long-Context Prefill Acceleration via Block-wise Dynamic Sparsification7 May
  2. 2025Tactic: Adaptive Sparse Attention with Clustering and Distribution Fitting for Long-Context LLMs17 Feb
  3. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
  4. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  5. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  6. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  7. 2023SparQ Attention: Bandwidth-Efficient LLM Inference8 Dec
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  9. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2025Qwen3 Technical Reportcited by 8,820
  6. 2020Longformer: The Long-Document Transformercited by 3,413
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  9. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  10. 2020Reformer: The Efficient Transformercited by 1,668
  11. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  12. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
21 more
  1. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  2. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  3. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 613
  4. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  5. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  6. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  7. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  8. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 198
  9. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 197
  10. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 196
  11. 2025RWKV-7 "Goose" with Expressive Dynamic State Evolutioncited by 141
  12. 2025XAttention: Block Sparse Attention with Antidiagonal Scoringcited by 137
  13. 2020Fast Transformers with Clustered Attentioncited by 123
  14. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 123
  15. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decodingcited by 102
  16. 2026FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scalingcited by 66
  17. 2025InfLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long Adaptationcited by 32
  18. 2025Rectifying Magnitude Neglect in Linear Attentioncited by 16
  19. 2025Optimizing Mixture of Block Attentioncited by 9
  20. 2026FlashPrefill: Instantaneous Pattern Discovery and Thresholding for Ultra-Fast Long-Context Prefillingcited by 9
  21. 2026Sol-Attn: Accelerating Video Generation Inference via On-the-Fly Attention Sparsificationcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026CompKV: Compensation-Aware KV Selection for Long-Context LLM Inferencecited by 1
  2. 2026SparseEngine: Sparse-First Inference Enginecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SALE : Low-bit Estimation for Efficient Sparse Attention in Long-context LLM Prefilling
  2. 2025Progressive Sparse Attention: Algorithm and System Co-design for Efficient Attention in LLM Serving
  3. 2026FAST-Prefill: FPGA Accelerated Sparse Attention for Long Context LLM Prefill ↗
  4. 2025LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention
  5. 2026FlashMemory-DeepSeek-V4: Lightning Index Ultra-Long Context via Lookahead Sparse Attention
  6. 2025Kascade: A Practical Sparse Attention Method for Long-Context LLM Inference
  7. 2025SlimInfer: Accelerating Long-Context LLM Inference via Dynamic Token Pruning
  8. 2026SPLASH: Co-Designing Sparse Attention with High-Bandwidth Flash for Efficient Long-Context Inference