# S3-Attention:Attention-Aligned Endogenous Retrieval for Memory-Bounded Long-Context Inference

arXiv: https://arxiv.org/abs/2601.17702 · HTML: https://mlsanity.com/paper/2601.17702/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 47 references that are in the map.
- First posted on arXiv 25 Jan 2026.

## Builds on (most cited first, 30 of 42)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023)
- [SnapKV: LLM Knows What You are Looking for Before Generation](https://mlsanity.com/paper/2404.14469/) (2024)
- [H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models](https://mlsanity.com/paper/2306.14048/) (2023)
- [Scaling and evaluating sparse autoencoders](https://mlsanity.com/paper/2406.04093/) (2024)
- [Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference](https://mlsanity.com/paper/2406.10774/) (2024)
- [LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens](https://mlsanity.com/paper/2402.13753/) (2024)
- [Benchmarking Large Language Models in Retrieval-Augmented Generation](https://mlsanity.com/paper/2309.01431/) (2023)
- [DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads](https://mlsanity.com/paper/2410.10819/) (2024)
- [Retrieval Head Mechanistically Explains Long-Context Factuality](https://mlsanity.com/paper/2404.15574/) (2024)
- [Landmark Attention: Random-Access Infinite Context Length for Transformers](https://mlsanity.com/paper/2305.16300/) (2023)
- [ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference](https://mlsanity.com/paper/2410.21465/) (2024)
- [A Comprehensive Survey on Long Context Language Modeling](https://mlsanity.com/paper/2503.17407/) (2025)
- [RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval](https://mlsanity.com/paper/2409.10516/) (2024)
- [ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference](https://mlsanity.com/paper/2502.00299/) (2025)
- [Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformers](https://mlsanity.com/paper/2406.16747/) (2024)
- [LongAgent: Scaling Language Models to 128k Context through Multi-Agent Collaboration](https://mlsanity.com/paper/2402.11550/) (2024)
- [A Survey on Knowledge-Oriented Retrieval-Augmented Generation](https://mlsanity.com/paper/2503.10677/) (2025)
- [Query-Focused Retrieval Heads Improve Long-Context Reasoning and Re-ranking](https://mlsanity.com/paper/2506.09944/) (2025)
- [LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs -- No Silver Bullet for LC or RAG Routing](https://mlsanity.com/paper/2502.09977/) (2025)
- [Tactic: Adaptive Sparse Attention with Clustering and Distribution Fitting for Long-Context LLMs](https://mlsanity.com/paper/2502.12216/) (2025)
- [HopRAG: Multi-Hop Reasoning for Logic-Aware Retrieval-Augmented Generation](https://mlsanity.com/paper/2502.12442/) (2025)
- [SCOPE: Optimizing Key-Value Cache Compression in Long-context Generation](https://mlsanity.com/paper/2412.13649/) (2024)
- [LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference](https://mlsanity.com/paper/2503.08879/) (2025)
- [On the Vulnerability of Applying Retrieval-Augmented Generation within Knowledge-Intensive Application Domains](https://mlsanity.com/paper/2409.17275/) (2024)
- [Multipole Attention for Efficient Long Context Reasoning](https://mlsanity.com/paper/2506.13059/) (2025)
- [Core Context Aware Transformers for Long Context Language Modeling](https://mlsanity.com/paper/2412.12465/) (2024)
- [Internal Causal Mechanisms Robustly Predict Language Model Out-of-Distribution Behaviors](https://mlsanity.com/paper/2505.11770/) (2025)
- [Understanding Synthetic Context Extension via Retrieval Heads](https://mlsanity.com/paper/2410.22316/) (2024)

## Built on by (most cited first, 1 of 1)

- [Connecting the Dots: Benchmarking Reflective Memory in Long-Horizon Dialogue](https://mlsanity.com/paper/2606.01223/) (2026)

## Abstract (arXiv)

Large language models are increasingly applied to multi-document and long-form inputs, yet long-context inference remains memory- and noise-inefficient. Key-value (KV) caching scales linearly with context length, while external retrieval methods often return lexically similar but causally irrelevant passages. We present S3-Attention, a memory-first inference-time framework that treats long-context processing as attention-aligned endogenous retrieval. S3-Attention decodes transient key and query projections into top-k sparse feature identifiers using lightweight sparse autoencoders, and constructs a CPU-based inverted index mapping features to token positions or spans during a single streaming scan. This design allows the KV cache to be discarded entirely and bounds GPU memory usage by the scan chunk size. At generation time, feature co-activation is used to retrieve compact evidence spans, optionally fused with BM25 for exact lexical matching. Under a unified LongBench evaluation protocol with fixed prompting, decoding, and matched token budgets, S3-Hybrid closely matches full-context inference across multiple model families and improves robustness in several information-dense settings. We also report an engineering limitation of the current prototype, which incurs higher wall-clock latency than optimized full-KV baselines, motivating future kernel-level optimization.