mlsanityData to 7 Oct 2026

S3-Attention:Attention-Aligned Endogenous Retrieval for Memory-Bounded Long-Context Inference

arXiv 2601.17702 ↗First posted 25 Jan 2026cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Large language models are increasingly applied to multi-document and long-form inputs, yet long-context inference remains memory- and noise-inefficient. Key-value (KV) caching scales linearly with context length, while external retrieval methods often return lexically similar but causally irrelevant passages. We present S3-Attention, a memory-first inference-time framework that treats long-context processing as attention-aligned endogenous retrieval. S3-Attention decodes transient key and query projections into top-k sparse feature identifiers using lightweight sparse autoencoders, and constructs a CPU-based inverted index mapping features to token positions or spans during a single streaming scan. This design allows the KV cache to be discarded entirely and bounds GPU memory usage by the scan chunk size. At generation time, feature co-activation is used to retrieve compact evidence spans, optionally fused with BM25 for exact lexical matching. Under a unified LongBench evaluation protocol with fixed prompting, decoding, and matched token budgets, S3-Hybrid closely matches full-context inference across multiple model families and improves robustness in several information-dense settings. We also report an engineering limitation of the current prototype, which incurs higher wall-clock latency than optimized full-KV baselines, motivating future kernel-level optimization.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 42 of its 47 references that are in the map. First posted on arXiv 25 Jan 2026.

Builds on

42 of 47 references.

References · 42

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  3. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  4. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  5. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  6. 2024Scaling and evaluating sparse autoencoderscited by 530
  7. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 487
  8. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 307
  9. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 302
  10. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 283
  11. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 219
  12. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
30 more
  1. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 132
  2. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 114
  3. 2025A Comprehensive Survey on Long Context Language Modelingcited by 113
  4. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inferencecited by 57
  5. 2024Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformerscited by 51
  6. 2024LongAgent: Scaling Language Models to 128k Context through Multi-Agent Collaborationcited by 38
  7. 2025A Survey on Knowledge-Oriented Retrieval-Augmented Generationcited by 34
  8. 2025Query-Focused Retrieval Heads Improve Long-Context Reasoning and Re-rankingcited by 30
  9. 2025LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs -- No Silver Bullet for LC or RAG Routingcited by 29
  10. 2025Tactic: Adaptive Sparse Attention with Clustering and Distribution Fitting for Long-Context LLMscited by 24
  11. 2025HopRAG: Multi-Hop Reasoning for Logic-Aware Retrieval-Augmented Generationcited by 24
  12. 2024SCOPE: Optimizing Key-Value Cache Compression in Long-context Generationcited by 20
  13. 2025LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inferencecited by 18
  14. 2024On the Vulnerability of Applying Retrieval-Augmented Generation within Knowledge-Intensive Application Domainscited by 14
  15. 2025Multipole Attention for Efficient Long Context Reasoningcited by 14
  16. 2024Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similaritycited by 11
  17. 2024Core Context Aware Transformers for Long Context Language Modelingcited by 11
  18. 2025Internal Causal Mechanisms Robustly Predict Language Model Out-of-Distribution Behaviorscited by 11
  19. 2024Understanding Synthetic Context Extension via Retrieval Headscited by 10
  20. 2025C-3PO: Compact Plug-and-Play Proxy Optimization to Achieve Human-like Retrieval-Augmented Generationcited by 9
  21. 2024Efficient Length-Generalizable Attention via Causal Retrieval for Long-Context Language Modelingcited by 8
  22. 2025LLM Alignment as Retriever Optimization: An Information Retrieval Perspectivecited by 7
  23. 2025In-context denoising with one-layer transformers: connections between attention and associative memory retrievalcited by 7
  24. 2025Infinite Retrieval: Attention Enhanced LLMs in Long-Context Processingcited by 7
  25. 2025LIFT: A Novel Framework for Enhancing Long-Context Understanding of LLMs via Long Input Fine-Tuningcited by 6
  26. 2025Re-ranking Reasoning Context with Tree Search Makes Large Vision-Language Models Strongercited by 6
  27. 2025POQD: Performance-Oriented Query Decomposer for Multi-vector retrievalcited by 5
  28. 2025DELTA: Dynamic Layer-Aware Token Attention for Efficient Long-Context Reasoningcited by 5
  29. 2025RAG+: Enhancing Retrieval-Augmented Generation with Application-Aware Reasoningcited by 3
  30. 2025Long-Short Alignment for Effective Long-Context Modeling in LLMscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps
  2. 2026Simplified Sparse Attention via Gist Tokens
  3. 2026OVAL: Output-Aware Local Page Bases for KV Cache Retrieval
  4. 2025Training-free Context-adaptive Attention for Efficient Long Context Modeling
  5. 2025RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inference
  6. 2025Retrospective Sparse Attention for Efficient Long-Context Generation
  7. 2026Sparse Attention as a Range Searching Problem: Towards an Inference-Efficient Index for KV Cache
  8. 2025Every Token Counts: Generalizing 16M Ultra-Long Context in Large Language Models