mlsanityData to 7 Oct 2026

SLA2: Sparse-Linear Attention with Learnable Routing and QAT

arXiv 2602.12675 ↗First posted 13 Feb 2026cs.LG cs.AI cs.CVcited by 21

On the map: Diffusion models → Video generation

Abstract (arXiv)

Sparse-Linear Attention (SLA) combines sparse and linear attention to accelerate diffusion models and has shown strong performance in video generation. However, (i) SLA relies on a heuristic split that assigns computations to the sparse or linear branch based on attention-weight magnitude, which can be suboptimal. Additionally, (ii) after formally analyzing the attention error in SLA, we identify a mismatch between SLA and a direct decomposition into sparse and linear attention. We propose SLA2, which introduces (I) a learnable router that dynamically selects whether each attention computation should use sparse or linear attention, (II) a more faithful and direct sparse-linear attention formulation that uses a learnable ratio to combine the sparse and linear attention branches, and (III) a sparse + low-bit attention design, where low-bit attention is introduced via quantization-aware fine-tuning to reduce quantization error. Experiments show that on video diffusion models, SLA2 can achieve 97% attention sparsity and deliver an 18.6x attention speedup while preserving generation quality.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 62 of its 65 references that are in the map. First posted on arXiv 13 Feb 2026.

Builds on

62 of 65 references.

Closest ancestors · 19

Same direction, newest first

  1. 2026Quant VideoGen: Auto-Regressive Long Video Generation via 2-Bit KV-Cache Quantization3 Feb
  2. 2025Trainable Log-linear Sparse Attention for Efficient Diffusion Transformers18 Dec
  3. 2025TurboDiffusion: Accelerating Video Diffusion Models by 100-200 Times18 Dec
  4. 2025UltraViCo: Breaking Extrapolation Limits in Video Diffusion Transformers25 Nov
  5. 2025SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse-Linear Attention28 Sep
  6. 2025Bidirectional Sparse Attention for Faster Video Diffusion Training1 Sep
  7. 2025Radial Attention: O(nlog n) Sparse Attention with Energy Decay for Long Video Generation24 Jun
  8. 2025FPSAttention: Training-Aware FP8 and Sparsity Co-Design for Fast Video Diffusion5 Jun
  9. 2025Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate Video Diffusion Transformers3 Jun
  10. 2025Re-ttention: Ultra Sparse Visual Generation via Attention Statistical Reshape28 May
  11. 2025Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutation24 May
  12. 2025VORTA: Efficient Video Diffusion via Routing Sparse Attention24 May
7 more
  1. 2025VSA: Faster Video Diffusion with Trainable Sparse Attention19 May
  2. 2025Training-free and Adaptive Sparse Attention for Efficient Long Video Generation28 Feb
  3. 2025DSV: Exploiting Dynamic Sparsity to Accelerate Large-Scale Video DiT Training11 Feb
  4. 2025Fast Video Generation with Sliding Tile Attention6 Feb
  5. 2025Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity3 Feb
  6. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer12 Aug
  7. 2024DiG: Scalable and Efficient Diffusion Models with Gated Linear Attention28 May

Other references · 43

  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  2. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  3. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,536
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  5. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  6. 2020Rethinking Attention with Performerscited by 1,584
  7. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  8. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  9. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 755
  10. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  11. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  12. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 487
31 more
  1. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  2. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  3. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  4. 2024SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformerscited by 331
  5. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 201
  6. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 188
  7. 2024SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Accelerationcited by 158
  8. 2024VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generationcited by 148
  9. 2025XAttention: Block Sparse Attention with Antidiagonal Scoringcited by 131
  10. 2024SeerAttention: Learning Intrinsic Sparse Attention in Your LLMscited by 121
  11. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 121
  12. 2023SparQ Attention: Bandwidth-Efficient LLM Inferencecited by 119
  13. 2022Overcoming Oscillations in Quantization-Aware Trainingcited by 116
  14. 2024SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantizationcited by 113
  15. 2025SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inferencecited by 113
  16. 2025Mixture of Contexts for Long Video Generationcited by 90
  17. 2025Large Scale Diffusion Distillation via Score-Regularized Continuous-Time Consistencycited by 81
  18. 2025SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Trainingcited by 63
  19. 2024Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Modelscited by 57
  20. 2025Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruningcited by 48
  21. 2024Post-Training Sparse Attention with Double Sparsitycited by 45
  22. 2024Evaluation Agent: Efficient and Promptable Evaluation Framework for Visual Generative Modelscited by 41
  23. 2025Tactic: Adaptive Sparse Attention with Clustering and Distribution Fitting for Long-Context LLMscited by 24
  24. 2025SageAttention2++: A More Efficient Implementation of SageAttention2cited by 21
  25. 2025Accurate INT8 Training Through Dynamic Block-Level Fallbackcited by 14
  26. 2026Geometry-Aware Rotary Position Embedding for Consistent Video World Modelcited by 12
  27. 2025Identifying Sensitive Weights via Post-quantization Integralcited by 10
  28. 2025UltraImage: Rethinking Resolution Extrapolation in Image Diffusion Transformerscited by 10
  29. 2026Residual Context Diffusion Language Modelscited by 9
  30. 2025SAGE: A Framework of Precise Retrieval for RAGcited by 8
  31. 2024Separate, Dynamic and Differentiable (SMART) Pruner for Block/Output Channel Pruning on Computer Vision Taskscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 21.

21 papers.

Sort:

By area:

  1. 2026Causal-rCM: A Unified Teacher-Forcing and Self-Forcing Open Recipe for Autoregressive Diffusion Distillation in Streaming Video Generation and Interactive World Modelscited by 23
  2. 2026Mode Seeking meets Mean Seeking for Fast Long Video Generationcited by 12
  3. 2026KernelBenchX: A Comprehensive Benchmark for Evaluating LLM-Generated GPU Kernelscited by 10
  4. 2026SVG-EAR: Parameter-Free Linear Compensation for Sparse Video Generation via Error-aware Routingcited by 9
  5. 2026SageBwd: A Trainable Low-bit Attentioncited by 7
  6. 2026Sol-Attn: Accelerating Video Generation Inference via On-the-Fly Attention Sparsificationcited by 6
  7. 2026MeMix: Writing Less, Remembering More for Streaming 3D Reconstructioncited by 5
  8. 2026Vidu S1: A Real-Time Interactive Video Generation Modelcited by 4
  9. 2026Vidu S2: Real-Time Interactive, Editable, and Spatial Video Generationcited by 4
  10. 2026HybridStitch: Pixel and Timestep Level Model Stitching for Diffusion Accelerationcited by 2
  11. 2026Ride the Wave: Precision-Allocated Sparse Attention for Smooth Video Generationcited by 2
  12. 2026OSP-Next: Efficient High-Quality Video Generation with Sparse Sequence Parallelism, HiF8 Quantization, and Reinforcement Learningcited by 1
9 more
  1. 2026Multi-Mixer Models: Flexible Sequence Modeling with Shared Representationscited by 1
  2. 2026HyperVAttention: Efficient Sparse Attention with Spatio-Temporal Clustering for Video Diffusioncited by 1
  3. 2026RoLA: Rotary-Positioned Low-Rank Linear Attention for Efficient Diffusion Transformerscited by 1
  4. 2026Video DeltaNet: A Video-Native Hybrid Attention for Livestream Video Generationcited by 1
  5. 20266Bit-Diffusion: Inference-Time Mixed-Precision Quantization for Video Diffusion Modelscited by 0
  6. 2026Flux Attention: Context-Aware Hybrid Attention for Efficient LLMs Inferencecited by 0
  7. 2026SparSTAR: Sparse Attention for SpaceTime AutoRegressive Video Synthesiscited by 0
  8. 2026SparkDiffusion: Mitigating the High-Sparsity Trap --- A Unified Framework for up to 265× Single-GPU Acceleration of Visual Generationcited by 0
  9. 2026iCATS: Fast Video Generation via Interaction-Aware Sparse Attention and Timestep-Adaptive Sparsitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026LoSA: Near-Lossless Sparse Attention for Training-Free Video Diffusion Acceleration
  2. 2026LVSA: Training-Free Sparse Attention for Long Video Diffusion
  3. 2026SALAD: Achieve High-Sparsity Attention via Efficient Linear Attention Tuning for Video Diffusion Transformer
  4. 2026SPADE: An Input-Adaptive Sparse Attention Engine for Fast Video Diffusion Models Inference
  5. 2026PISA: Piecewise Sparse Attention Is Wiser for Efficient Diffusion Transformers
  6. 2025Attention Surgery: An Efficient Recipe to Linearize Your Video Diffusion Transformer
  7. 2026Veda: Scalable Video Diffusion via Distilled Sparse Attention
  8. 2026SANA-Video 2.0: Hybrid Linear Attention with Attention Residuals for Efficient Video Generation