mlsanityData to 14 Oct 2026

Models Take Notes at Prefill: KV Cache Can Be Editable and Composable

arXiv 2606.17107 ↗First posted 14 Jun 2026cs.LG cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

Prefix caching reuses prefill only across an exactly shared prefix, so one changed field invalidates the entire downstream cache. Yet overwriting the field's own key/value vectors and reusing the rest leaves the model acting on the old value. The reason, established causally across four model families: at prefill the model has already written the field-conditioned conclusion onto downstream notes; the field's own key/value drives under 1% of the decision. Read as a notebook of memoized conclusions, two capabilities follow. (1) It is editable. A salient erratum amends the notes; and with chain-of-thought, editing the field alone recovers the decision (1.00 at 8B, ~1% compute), while without CoT it is ignored. (2) It is composable. The notes are position-portable, so a precompiled skill can be RoPE-repositioned and spliced into any context, indistinguishable from full recompute (logit cosine 0.90-0.999, twelve models) at O(L) rather than O(L^2) time-to-first-token. A unified edit+compose agent stays decision-identical to recompute at up to 14.9x lower latency. The approach applies to any per-token attention KV cache, validated across scale, quantization, Mixture-of-Experts, and multimodal caches, and extends to several attention variants through small adapters. Because the erratum is append-only, it composes with production prefix caching: in an online vLLM benchmark it keeps the prefix cache-aligned (98.5% hit-rate), cutting p90 time-to-first-token by 53-398x.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 33 of its 38 references that are in the map. First posted on arXiv 14 Jun 2026.

Builds on

33 of 38 references.

References · 33

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  7. 2023Mistral 7Bcited by 3,164
  8. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  9. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  10. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  11. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  12. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
21 more
  1. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  2. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  3. 2022Editing Models with Task Arithmeticcited by 1,120
  4. 2022Mass-Editing Memory in a Transformercited by 1,001
  5. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  6. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  7. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 830
  8. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  9. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 815
  10. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 627
  11. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  12. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
  13. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 439
  14. 2023Function Vectors in Large Language Modelscited by 308
  15. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 268
  16. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 223
  17. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 203
  18. 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 98
  19. 2025KVLink: Accelerating Large Language Models via Efficient KV Cache Reusecited by 66
  20. 2024EPIC: Efficient Position-Independent Caching for Serving Large Language Modelscited by 47
  21. 2025MPIC: Position-Independent Multimodal Context Caching System for Efficient MLLM Servingcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026Can I Buy Your KV Cache?
  2. 2026Elastic KV Cache for LLM Serving:A Working Reclamation Mechanism, and Why Chunked Prefill Already Closes the Gap
  3. 2026What It Costs to Compose, Rebuild, and Correct Precomputed Memory
  4. 2026KVShareArena: KV-Cache Reuse Across Contexts and Model Checkpoints
  5. 2026Written as a Record, Read as an Address: What a Forward Pass Leaves in an Operation's KV Cache
  6. 2025EFIM: Efficient Serving of LLMs for Infilling Tasks with Improved KV Cache Reuse
  7. 2025KV Cache Recycling to Expand Usable Context Capacity in Low Parameter LLMs
  8. 2026PAGE: Partition-Aware Gated KV-Cache Eviction