mlsanityData to 7 Oct 2026

Nectar: Neural Estimation of Cached-Token Attention via Regression

arXiv 2605.09778 ↗First posted 10 May 2026cs.LG cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

Evaluating softmax attention over a fixed long context requires reading every cached key-value pair for each new query token. For a given context (a book, a manual, a legal corpus) the attention output is a deterministic function of the query. We propose Nectar, which fits a compact neural network to this function for queries drawn from a task-relevant distribution. Nectar fits two networks per layer and KV-head: a target network that predicts the attention output and a score network that predicts the log-normalizer. The pair plugs into the standard masked self-attention at inference time, replacing the O(n) attention over the cache with a forward pass whose cost does not depend on n. Each module carries on the order of |θ| parameters per layer and KV-head, typically much smaller than the 2nd KV-cache footprint at the same granularity. We report experiments on models from 1.7B to 8B parameters across five long-context datasets. The approximation error tracks the next-token accuracy gap to full attention, and allocating capacity non-uniformly across layers reduces that gap in our ablation. Beyond this analysis of metrics, we check that the text generations (following a question prompt) of a model equipped with a Nectar module match in semantic content those obtained by giving the same model access to the full cache.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 30 of its 36 references that are in the map. First posted on arXiv 10 May 2026.

Builds on

30 of 36 references.

References · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2025Qwen3 Technical Reportcited by 8,195
  7. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  9. 2020Longformer: The Long-Document Transformercited by 3,362
  10. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  11. 2020Reformer: The Efficient Transformercited by 1,658
  12. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
18 more
  1. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  2. 2025OpenAI GPT-5 System Cardcited by 910
  3. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  4. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  5. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  6. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  7. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  8. 2022Efficiently Scaling Transformer Inferencecited by 529
  9. 2016Input Convex Neural Networkscited by 407
  10. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  11. 2023Adapting Language Models to Compress Contextscited by 350
  12. 2023Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMscited by 175
  13. 2025Qwen2.5-1M Technical Reportcited by 158
  14. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 74
  15. 2025Text-to-LoRA: Instant Transformer Adaptioncited by 50
  16. 2024LongHealth: A Question Answering Benchmark with Long Clinical Documentscited by 33
  17. 2025Training Plug-n-Play Knowledge Modules with Deep Context Distillationcited by 22
  18. 2026Amortized Maximum Inner Product Search with Learned Support Functionscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Cartridges++: KV Cache Compression without Off-Context Derailmentcited by 1
  2. 2026KV-Kaizen: Learning Context-Adaptive Cache Compression Choicescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Neural Attention Search
  2. 2026IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse
  3. 2026Simplified Sparse Attention via Gist Tokens
  4. 2026Augmenting Attention with Exponentially Decaying Memory Improves Query-Aware KV Sparsity
  5. 2026From Rigid to Dynamic: Entropy-Guided Adaptive Inference for Long-Context LLMs
  6. 2026ResKV: Reconstructing Omitted Attention Contributions for Fixed-Budget KV Cache Compression
  7. 2025Tactic: Adaptive Sparse Attention with Clustering and Distribution Fitting for Long-Context LLMs
  8. 2026Sparse Attention as a Range Searching Problem: Towards an Inference-Efficient Index for KV Cache