mlsanityData to 14 Oct 2026

Written as a Record, Read as an Address: What a Forward Pass Leaves in an Operation's KV Cache

arXiv 2609.24635 ↗First posted 21 Sep 2026cs.CLcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

When a language model reads an operation such as "Swap the contents of Box F and Box B", its forward pass writes keys and values for those tokens into the KV cache. Prior work on entity tracking establishes what models use: bindings are resolved at query time rather than stored as explicit latent state. We ask what they write at the operation span and how it is accessed. We split a forward pass into a frozen writer and a reader: the writer's cache is recomputed without gradients, while the reader sees only the instruction and operation tokens, with all state descriptions hidden, and is trained in isolation. Anything the reader recovers was therefore already present in the unmodified cache. On a synthetic boxes task, a base reader recovers ≤ 0.06 of queried bindings against 0.75--1.00 after training, and recoverability tracks the operation's read/write footprint. We find two modes of access. Across Llama-3.1-8B and Mistral-7B, operation-span transplants causally redirect which visible state is read even when the two worlds hold identical values, revealing a routing record. Isolation training preserves routing and adds direct access to the payload, the value the operation read, from the single operand-name token in a narrow mid-depth band (layers 12--15 of 32 in Llama-3.1-8B, 14--17 in Mistral-7B) --- the same site that holds the routing record. The same recipe extends to further operations, ToMi and GSM8K, but is bounded by training coverage and costs open-book accuracy. Operation tokens thus leave localized, causally recoverable records that support both routing and direct payload access, though the model that writes them reads mainly the address they carry and not the value.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 26 of its 30 references that are in the map. First posted on arXiv 21 Sep 2026.

Builds on

26 of 30 references.

References · 26

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2024Qwen2.5 Technical Reportcited by 4,942
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  6. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  7. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  8. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  9. 2024Yi: Open Foundation Models by 01.AIcited by 719
  10. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  11. 2019Designing and Interpreting Probes with Control Taskscited by 640
  12. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 528
14 more
  1. 2021Causal Abstractions of Neural Networkscited by 422
  2. 2023Learning to Compress Prompts with Gist Tokenscited by 390
  3. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 359
  4. 2024Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Modelscited by 235
  5. 2021Implicit Representations of Meaning in Neural Language Modelscited by 180
  6. 2024Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Trackingcited by 149
  7. 2023How do Language Models Bind Entities in Context?cited by 108
  8. 2023Entity Tracking in Language Modelscited by 44
  9. 2025Language Models use Lookbacks to Track Beliefscited by 39
  10. 2026Do Language Models Track Entities Across State Changes?cited by 6
  11. 2026KV-Skill: Forging Expertise in the Model's Native Languagecited by 4
  12. 2026A retrieval conditioned rebinding circuit for dynamic entity tracking in large language modelscited by 3
  13. 2026When Does Activation Steering Change What a Model Computes From?cited by 2
  14. 2026SCIT: Testing Causal Cache Carriers in Latent Chain-of-Thought Modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Positions Are Not Facts: The Mismatch Between KV Caches and Memory
  2. 2026Knowing the Store: What a Memory Backend Must Write Down Before an Agent Can Read It
  3. 2026Working Around the Compute Ceiling: Byte-Exact Memory in Galahad Makes LLM Reading a One-Time Cost LLM Reading a One-Time Cost
  4. 2025Cache Me If You Can: How Many KVs Do You Need for Effective Long-Context LMs?
  5. 2026Aborted but Not Forgotten: KV-Cache Retention Breaks Rollback Consistency in Language Agents
  6. 2025KV Cache Recycling to Expand Usable Context Capacity in Low Parameter LLMs
  7. 2026What It Costs to Compose, Rebuild, and Correct Precomputed Memory
  8. 2026Thinking Outside the Box: Retention and Transmission of Information in Sliding-Window KV Inference