mlsanityData to 7 Oct 2026

SSA: Sparse Sparse Attention by Aligning Full and Sparse Attention Outputs in Feature Space

arXiv 2511.20102 ↗First posted 25 Nov 2025cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Sparse attention reduces the quadratic complexity of full self-attention but faces two challenges: (1) an attention gap, where applying sparse attention to full-attention-trained models causes performance degradation due to train-inference distribution mismatch, and (2) a capability gap, where models trained purely with sparse attention lack complete gradient flow, preventing them from matching full-attention performance. We propose SSA (Sparse Sparse Attention), a training framework that integrates both sparse and full attention with bidirectional attention-output alignment. We prove that the approximation error scales linearly with the attention mass dropped under sparse attention, and show that SSA's alignment objective substantially reduces this quantity compared to baselines. Experiments demonstrate that SSA achieves state-of-the-art performance under both inference modes, adapts smoothly to varying sparsity budgets, and demonstrates superior long-context capabilities.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 37 of its 46 references that are in the map. First posted on arXiv 25 Nov 2025.

Builds on

37 of 46 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2017Neural Discrete Representation Learningcited by 4,650
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2020Longformer: The Long-Document Transformercited by 3,362
  8. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  9. 2016Pointer Sentinel Mixture Modelscited by 2,992
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  11. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  12. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
25 more
  1. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  2. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  3. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  4. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  5. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  6. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 487
  7. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  8. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 381
  9. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  10. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 357
  11. 2025DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environmentscited by 268
  12. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 266
  13. 2025CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillationcited by 216
  14. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 192
  15. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 188
  16. 2019Sparse Sequence-to-Sequence Modelscited by 167
  17. 2025XAttention: Block Sparse Attention with Antidiagonal Scoringcited by 131
  18. 2024Liger Kernel: Efficient Triton Kernels for LLM Trainingcited by 123
  19. 2024SeerAttention: Learning Intrinsic Sparse Attention in Your LLMscited by 121
  20. 2025SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inferencecited by 113
  21. 2020Shortformer: Better Language Modeling using Shorter Inputscited by 76
  22. 2025InfLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long Adaptationcited by 31
  23. 2025SeerAttention-R: Sparse Attention Adaptation for Long Reasoningcited by 28
  24. 2025Efficient Attention Mechanisms for Large Language Models: A Surveycited by 21
  25. 2025AdaSplash: Adaptive Sparse Flash Attentioncited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reusecited by 27
  2. 2026CoSA: Accelerating Long-Context Inference via Proxy-Kernel Co-Designed Sparse Attentioncited by 1
  3. 2026Trust the Mass: Forced Weights in KV-Cache Evictioncited by 1
  4. 2026Pre-registered tests of solid-state-physics-inspired LLM compression: a cluster-level negative result at small-language-model scalecited by 0