# Accelerating Prefilling for Long-Context LLMs via Sparse Pattern Sharing

arXiv: https://arxiv.org/abs/2505.19578 · HTML: https://mlsanity.com/paper/2505.19578/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 14 references that are in the map.
- First posted on arXiv 26 May 2025.

## Builds on (most cited first, 9 of 9)

- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [Compressive Transformers for Long-Range Sequence Modelling](https://mlsanity.com/paper/1911.05507/) (2019)
- [MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention](https://mlsanity.com/paper/2407.02490/) (2024)
- [∞Bench: Extending Long Context Evaluation Beyond 100K Tokens](https://mlsanity.com/paper/2402.13718/) (2024)
- [Multi-agent Architecture Search via Agentic Supernet](https://mlsanity.com/paper/2502.04180/) (2025)
- [Knowledge Graph Prompting for Multi-Document Question Answering](https://mlsanity.com/paper/2308.11730/) (2023)
- [CodePlan: Repository-level Coding using LLMs and Planning](https://mlsanity.com/paper/2309.12499/) (2023)
- [Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance](https://mlsanity.com/paper/2305.17306/) (2023)

## Built on by (most cited first, 4 of 4)

- [ProxyAttn: Guided Sparse Attention via Representative Heads](https://mlsanity.com/paper/2509.24745/) (2025)
- [Elastic Attention: Test-time Adaptive Sparsity Ratios for Efficient Transformers](https://mlsanity.com/paper/2601.17367/) (2026)
- [EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction](https://mlsanity.com/paper/2603.22910/) (2026)
- [Flux Attention: Context-Aware Hybrid Attention for Efficient LLMs Inference](https://mlsanity.com/paper/2604.07394/) (2026)

## Abstract (arXiv)

Sparse attention methods exploit the inherent sparsity in attention to speed up the prefilling phase of long-context inference, mitigating the quadratic complexity of full attention computation. While existing sparse attention methods rely on predefined patterns or inaccurate estimations to approximate attention behavior, they often fail to fully capture the true dynamics of attention, resulting in reduced efficiency and compromised accuracy. Instead, we propose a highly accurate sparse attention mechanism that shares similar yet precise attention patterns across heads, enabling a more realistic capture of the dynamic behavior of attention. Our approach is grounded in two key observations: (1) attention patterns demonstrate strong inter-head similarity, and (2) this similarity remains remarkably consistent across diverse inputs. By strategically sharing computed accurate patterns across attention heads, our method effectively captures actual patterns while requiring full attention computation for only a small subset of heads. Comprehensive evaluations demonstrate that our approach achieves superior or comparable speedup relative to state-of-the-art methods while delivering the best overall accuracy.