# Understanding Sparse Attention Selectivity in Long-Context Foundation Models via Counterfactual Evaluation

arXiv: https://arxiv.org/abs/2608.01676 · HTML: https://mlsanity.com/paper/2608.01676/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 29 references that are in the map.
- First posted on arXiv 3 Aug 2026.

## Builds on (most cited first, 21 of 21)

- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019)
- [SnapKV: LLM Knows What You are Looking for Before Generation](https://mlsanity.com/paper/2404.14469/) (2024)
- [H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models](https://mlsanity.com/paper/2306.14048/) (2023)
- [Towards Automated Circuit Discovery for Mechanistic Interpretability](https://mlsanity.com/paper/2304.14997/) (2023)
- [Attention is not not Explanation](https://mlsanity.com/paper/1908.04626/) (2019)
- [Fact or Fiction: Verifying Scientific Claims](https://mlsanity.com/paper/2004.14974/) (2020)
- [Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention](https://mlsanity.com/paper/2502.11089/) (2025)
- [Is Attention Interpretable?](https://mlsanity.com/paper/1906.03731/) (2019)
- [PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models](https://mlsanity.com/paper/2402.07867/) (2024)
- [Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution](https://mlsanity.com/paper/2510.00636/) (2025)
- [SCBench: A KV Cache-Centric Analysis of Long-Context Methods](https://mlsanity.com/paper/2412.10319/) (2024)
- [Towards More Robust Retrieval-Augmented Generation: Evaluating RAG Under Adversarial Poisoning Attacks](https://mlsanity.com/paper/2412.16708/) (2024)
- [Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction](https://mlsanity.com/paper/2605.09649/) (2026)
- [ReST-KV: Robust KV Cache Eviction with Layer-wise Output Reconstruction and Spatial-Temporal Smoothing](https://mlsanity.com/paper/2605.08840/) (2026)
- [Understanding the Physics of Key-Value Cache Compression for LLMs through Attention Dynamics](https://mlsanity.com/paper/2603.01426/) (2026)
- [PerspectiveGap: A Benchmark for Multi-Agent Orchestration Prompting](https://mlsanity.com/paper/2606.08878/) (2026)
- [Agon: An Autonomous Large-Scale Omnidisciplinary Research System Built on Prompt Economy](https://mlsanity.com/paper/2606.24177/) (2026)
- [A Preliminary Study on the Promises and Challenges of Native Top-k Sparse Attention](https://mlsanity.com/paper/2512.03494/) (2025)
- [Gated Sparse Attention: Combining Computational Efficiency with Training Stability for Long-Context Language Models](https://mlsanity.com/paper/2601.15305/) (2026)

## Built on by (most cited first, 1 of 1)

- [Counterexamples to Local Reconstruction Gain as a Proxy for Final Fidelity in Residual Completion](https://mlsanity.com/paper/2609.34063/) (2026)

## Abstract (arXiv)

Sparse attention is widely deployed in long-context serving stacks, yet no framework audits how discarding blocks changes the influence of specific content on model output. We first establish that the phenomenon is real and causal: Block Sparse Flash Attention (BSFA) route replay across four architectures changes output decisions in 13 of 16 cells, with zero identity-replay label flips. We then introduce a dense-calibrated counterfactual audit using matched probe cards---Gold (carrying the correct answer label), Poison (carrying a target wrong label), and Benign (filler only)---under six-layout position symmetry, isolating the sparsification-specific effect. Two patterns compete. Signal concentration: the selector preserves Gold and Poison blocks far above filler-matched Benign blocks (G≈PggB across all model--task pairs). Integration loss: discarding blocks severs cross-block attention---confirmed by an ablation where isolating the probe block collapses its influence from 4.48 logits to zero. Compression ratio governs the balance: a full sweep from mild (c=0.25) to aggressive (c=0.75) compression across four model--task pairs reveals that three of four cells move toward stronger sparse amplification at higher compression, with two exhibiting sign reversals. Three independent arms---BSFA route replay, controlled block-top-k, and KV-cache eviction---converge: sparsification changes content influence in ways aggregate accuracy cannot detect. We provide an open measurement framework deployable on any model exposing block identities.