Understanding Sparse Attention Selectivity in Long-Context Foundation Models via Counterfactual Evaluation
Not on the map. Why?
Abstract (arXiv)
Sparse attention is widely deployed in long-context serving stacks, yet no framework audits how discarding blocks changes the influence of specific content on model output. We first establish that the phenomenon is real and causal: Block Sparse Flash Attention (BSFA) route replay across four architectures changes output decisions in 13 of 16 cells, with zero identity-replay label flips. We then introduce a dense-calibrated counterfactual audit using matched probe cards---Gold (carrying the correct answer label), Poison (carrying a target wrong label), and Benign (filler only)---under six-layout position symmetry, isolating the sparsification-specific effect. Two patterns compete. Signal concentration: the selector preserves Gold and Poison blocks far above filler-matched Benign blocks (G≈PggB across all model--task pairs). Integration loss: discarding blocks severs cross-block attention---confirmed by an ablation where isolating the probe block collapses its influence from 4.48 logits to zero. Compression ratio governs the balance: a full sweep from mild (c=0.25) to aggressive (c=0.75) compression across four model--task pairs reveals that three of four cells move toward stronger sparse amplification at higher compression, with two exhibiting sign reversals. Three independent arms---BSFA route replay, controlled block-top-k, and KV-cache eviction---converge: sparsification changes content influence in ways aggregate accuracy cannot detect. We provide an open measurement framework deployable on any model exposing block identities.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 21 of its 29 references that are in the map. First posted on arXiv 3 Aug 2026.
Builds on
21 of 29 references.
References · 21
- 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
- 2019Attention is not Explanationcited by 916
- 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
- 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
- 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
- 2019Attention is not not Explanationcited by 629
- 2020Fact or Fiction: Verifying Scientific Claimscited by 546
- 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
- 2019Is Attention Interpretable?cited by 387
- 2024PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Modelscited by 251
- 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distributioncited by 70
9 more
- 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methodscited by 59
- 2024Towards More Robust Retrieval-Augmented Generation: Evaluating RAG Under Adversarial Poisoning Attackscited by 14
- 2026Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Evictioncited by 10
- 2026ReST-KV: Robust KV Cache Eviction with Layer-wise Output Reconstruction and Spatial-Temporal Smoothingcited by 9
- 2026PerspectiveGap: A Benchmark for Multi-Agent Orchestration Promptingcited by 6
- 2026Understanding the Physics of Key-Value Cache Compression for LLMs through Attention Dynamicscited by 5
- 2026Agon: An Autonomous Large-Scale Omnidisciplinary Research System Built on Prompt Economycited by 4
- 2025A Preliminary Study on the Promises and Challenges of Native Top-k Sparse Attentioncited by 3
- 2026Gated Sparse Attention: Combining Computational Efficiency with Training Stability for Long-Context Language Modelscited by 3
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Pattern Selectivity is Not Task-Causal Structure: A Cross-Architecture Mechanistic Study of Composed-Task Circuits in 1B-Class Language Models
- 2026AB-Sparse: Sparse Attention with Adaptive Block Size for Accurate and Efficient Long-Context Inference
- 2026FlashBlock: Attention Caching for Efficient Long-Context Block Diffusion
- 2026SPLA: Block Sparse Plus Linear Attention for Long Context Modeling
- 2024SeerAttention: Learning Intrinsic Sparse Attention in Your LLMs
- 2026SpargeAttention2: Trainable Sparse Attention via Hybrid Top-k+Top-p Masking and Distillation Fine-Tuning
- 2026Attention Sinks in Diffusion Transformers: A Causal Analysis
- 2025Mixture of Sparse Attention: Content-Based Learnable Sparse Attention via Expert-Choice Routing