mlsanityData to 14 Oct 2026

XAttention: Block Sparse Attention with Antidiagonal Scoring

arXiv 2503.16428 ↗First posted 20 Mar 2025cs.CL cs.CVcited by 137

Not on the map. Why?

Cited by 35 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Long-Context Transformer Models (LCTMs) are vital for real-world applications but suffer high computational costs due to attention's quadratic complexity. Block-sparse attention mitigates this by focusing computation on critical regions, yet existing methods struggle with balancing accuracy and efficiency due to costly block importance measurements. In this paper, we introduce XAttention, a plug-and-play framework that dramatically accelerates long-context inference in Transformers models using sparse attention. XAttention's key innovation is the insight that the sum of antidiagonal values (i.e., from the lower-left to upper-right) in the attention matrix provides a powerful proxy for block importance. This allows for precise identification and pruning of non-essential blocks, resulting in high sparsity and dramatically accelerated inference. Across comprehensive evaluations on demanding long-context benchmarks-including RULER and LongBench for language, VideoMME for video understanding, and VBench for video generation. XAttention achieves accuracy comparable to full attention while delivering substantial computational gains. We demonstrate up to 13.5x acceleration in attention computation. These results underscore XAttention's ability to unlock the practical potential of block sparse attention, paving the way for scalable and efficient deployment of LCTMs in real-world applications. Code is available at https://github.com/mit-han-lab/x-attention.

Key facts. Cited by 137 papers in the map (data through 14 Oct 2026). Builds on 26 of its 37 references that are in the map. First posted on arXiv 20 Mar 2025.

Builds on

26 of 37 references.

References · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Scalable Diffusion Models with Transformerscited by 6,054
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  5. 2020Longformer: The Long-Document Transformercited by 3,413
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  7. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  8. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  9. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  10. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  11. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  12. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
14 more
  1. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  2. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  3. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  4. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  5. 2023FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attentioncited by 313
  6. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 288
  7. 2025Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsitycited by 217
  8. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 208
  9. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 197
  10. 2025Fast Video Generation with Sliding Tile Attentioncited by 128
  11. 2024SeerAttention: Learning Intrinsic Sparse Attention in Your LLMscited by 124
  12. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 123
  13. 2024DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Modelscited by 98
  14. 2024Selective Attention Improves Transformercited by 26

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 52; 2026: 85.

137 papers.

Sort:

By area:

  1. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 196
  2. 2025Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutationcited by 106
  3. 2025VSA: Faster Video Diffusion with Trainable Sparse Attentioncited by 104
  4. 2025Radial Attention: O(nlog n) Sparse Attention with Energy Decay for Long Video Generationcited by 69
  5. 2025MiniCPM4: Ultra-Efficient LLMs on End Devicescited by 52
  6. 2025InfLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long Adaptationcited by 32
  7. 2026IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reusecited by 31
  8. 2025SeerAttention-R: Sparse Attention Adaptation for Long Reasoningcited by 30
  9. 2025MMInference: Accelerating Pre-filling for Long-Context VLMs via Modality-Aware Permutation Sparse Attentioncited by 26
  10. 2026Flow caching for autoregressive video generationcited by 25
  11. 2025DraftAttention: Fast Video Diffusion via Low-Resolution Attention Guidancecited by 22
  12. 2025Efficient Attention Mechanisms for Large Language Models: A Surveycited by 22
38 more
  1. 2026SLA2: Sparse-Linear Attention with Learnable Routing and QATcited by 22
  2. 2025Long-Context Generalization with Sparse Attentioncited by 21
  3. 2025Generalized Neighborhood Attention: Multi-dimensional Sparse Attention at the Speed of Lightcited by 18
  4. 2026SpargeAttention2: Trainable Sparse Attention via Hybrid Top-k+Top-p Masking and Distillation Fine-Tuningcited by 18
  5. 2026PISA: Piecewise Sparse Attention Is Wiser for Efficient Diffusion Transformerscited by 17
  6. 2025Comba: Improving Bilinear RNNs with Closed-loop Controlcited by 15
  7. 2025Mixing Importance with Diversity: Joint Optimization for KV Cache Compression in Large Vision-Language Modelscited by 14
  8. 2025RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inferencecited by 13
  9. 2025BLADE: Block-Sparse Attention Meets Step Distillation for Efficient Video Generationcited by 13
  10. 2025BLASST: Dynamic BLocked Attention Sparsity via Softmax Thresholdingcited by 13
  11. 2025Kinetics: Rethinking Test-Time Scaling Lawscited by 12
  12. 2026Light Forcing: Accelerating Autoregressive Video Diffusion via Sparse Attentioncited by 12
  13. 2026HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attentioncited by 12
  14. 2025MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systemscited by 11
  15. 2025VideoNSA: Native Sparse Attention Scales Video Understandingcited by 11
  16. 2025LinVideo: A Post-Training Framework towards O(n) Attention in Efficient Video Generationcited by 11
  17. 2026Sparse Forcing: Native Trainable Sparse Attention for Real-time Autoregressive Diffusion Video Generationcited by 11
  18. 2025FastCar: Cache Attentive Replay for Fast Auto-Regressive Video Generation on the Edgecited by 10
  19. 2025Re-ttention: Ultra Sparse Visual Generation via Attention Statistical Reshapecited by 10
  20. 2025UniGist: Towards General and Hardware-aligned Sequence-level Long Context Compressioncited by 9
  21. 2025ProxyAttn: Guided Sparse Attention via Representative Headscited by 9
  22. 2025Optimizing Mixture of Block Attentioncited by 9
  23. 2026FlashPrefill: Instantaneous Pattern Discovery and Thresholding for Ultra-Fast Long-Context Prefillingcited by 9
  24. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospectscited by 9
  25. 2026Sol-Attn: Accelerating Video Generation Inference via On-the-Fly Attention Sparsificationcited by 9
  26. 2025Chipmunk: Training-Free Acceleration of Diffusion Transformers with Dynamic Column-Sparse Deltascited by 8
  27. 2025SalQ-VLM: Fine-Grained Saliency-Guided Quantization for Vision-Language Modelscited by 8
  28. 2026Sol Video Inference Engine: Agent-Native Full-Stack Acceleration Framework for Efficient Video Generationcited by 8
  29. 2025FSA: An Alternative Efficient Implementation of Native Sparse Attention Kernelcited by 7
  30. 2025SpecPV: Improving Self-Speculative Decoding for Long-Context Generation via Partial Verificationcited by 7
  31. 2026Focus-dLLM: Accelerating Long-Context Diffusion LLM Inference via Confidence-Guided Context Focusingcited by 7
  32. 2026Efficient Video Diffusion Models: Advancements and Challengescited by 7
  33. 2025SSA: Sparse Sparse Attention by Aligning Full and Sparse Attention Outputs in Feature Spacecited by 6
  34. 2026AngelSlim: A more accessible, comprehensive, and efficient toolkit for large model compressioncited by 6
  35. 2026DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generationcited by 6
  36. 2025LOOM-Scope: a comprehensive and efficient LOng-cOntext Model evaluation frameworkcited by 5
  37. 2025Accelerating Prefilling via Decoding-time Contribution Sparsitycited by 5
  38. 2025Revisiting Long-context Modeling from Context Denoising Perspectivecited by 5

50 most cited of 137 All 137 (JSON)