mlsanityData to 14 Oct 2026

Neural Attention Search

arXiv 2502.13251 ↗First posted 18 Feb 2025cs.CL cs.AIcited by 1

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

We present Neural Attention Search (NAtS), a framework that automatically evaluates the importance of each token within a sequence and determines if the corresponding token can be dropped after several steps. This approach can efficiently reduce the KV cache sizes required by transformer-based models during inference and thus reduce inference costs. In this paper, we design a search space that contains three token types: (i) Global Tokens will be preserved and queried by all the following tokens. (ii) Local Tokens survive until the next global token appears. (iii) Sliding Window Tokens have an impact on the inference of a fixed size of the next following tokens. Similar to the One-Shot Neural Architecture Search approach, this token-type information can be learned jointly with the architecture weights via a learnable attention mask. Experiments on both training a new transformer from scratch and fine-tuning existing large language models show that NAtS can efficiently reduce the KV cache size required for the models while maintaining the models' performance.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 74 of its 94 references that are in the map. First posted on arXiv 18 Feb 2025.

Builds on

74 of 94 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025CriticalKV: Optimizing KV Cache Eviction from an Output Perturbation Perspective6 Feb
  2. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference1 Feb
  3. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving2 Jan
  4. 2024A Survey on Large Language Model Acceleration based on KV Cache Management27 Dec
  5. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference28 Oct
  6. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
  7. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  8. 2024D2O: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Models18 Jun
  9. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  10. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  11. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  12. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference14 Mar
4 more
  1. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  3. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun
  4. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 58

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  9. 2020Longformer: The Long-Document Transformercited by 3,413
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  11. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  12. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
38 more
  1. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  3. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  4. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  5. 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
  6. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  7. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  8. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  9. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
  10. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,148
  11. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,101
  12. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
  13. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
  14. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 837
  15. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  16. 2017The NarrativeQA Reading Comprehension Challengecited by 597
  17. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 535
  18. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 486
  19. 2023Benchmarking Large Language Models for News Summarizationcited by 445
  20. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  21. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  22. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  23. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 365
  24. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 353
  25. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compressioncited by 348
  26. 2021Efficient Attentions for Long Document Summarizationcited by 345
  27. 2019Searching for A Robust Neural Architecture in Four GPU Hourscited by 260
  28. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 258
  29. 2015Non-stochastic Best Arm Identification and Hyperparameter Optimizationcited by 221
  30. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 213
  31. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 197
  32. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 193
  33. 2024A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systemscited by 170
  34. 2021Learned Token Pruning for Transformerscited by 139
  35. 2023Neural Architecture Search: Insights from 1000 Paperscited by 139
  36. 2024SeerAttention: Learning Intrinsic Sparse Attention in Your LLMscited by 124
  37. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 116
  38. 2023LongCoder: A Long-Range Pre-trained Language Model for Code Completioncited by 112

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Neural Attention Search Linear: Towards Adaptive Token-Level Hybrid Attention Modelscited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2026Nectar: Neural Estimation of Cached-Token Attention via Regression
  2. 2025Is Random Attention Sufficient for Sequence Modeling? Disentangling Trainable Components in the Transformer
  3. 2024Taipan: Efficient and Expressive State Space Language Models with Selective Attention
  4. 2026Titan-Llama: Retrofitting Pre-trained LLMs with External Neural Memory Modules
  5. 2025ZeroTuning: Unlocking the Initial Token's Power to Enhance Large Language Models Without Training
  6. 2026Linear Predictability of Attention Heads in Large Language Models
  7. 2024RecurFormer: Not All Transformer Heads Need Self-Attention
  8. 2026Augmenting Attention with Exponentially Decaying Memory Improves Query-Aware KV Sparsity