mlsanityData to 7 Oct 2026

Real Long-Term Memory for AI: A 50-Million-Token Window That Is Faster and Cheaper Than Recompute

arXiv 2610.10845 ↗First posted 7 Oct 2026cs.CL cs.AI cs.DCcited by 0

Not on the map. Why?

Abstract (arXiv)

A large language model can only use the text that fits in its context window, and it recomputes its internal key-value (KV) state for a prompt every time the prompt is sent. We test a memory layer, the public package galahad-kv, that saves the KV state of each block of about 16,000 tokens to encrypted local NVMe disk and loads it back later, byte-exact, without recomputing it. We ran it on 50,000,000 tokens of real public text, served through vLLM on one NVIDIA H100, with Gemma 4 12B and Gemma 4 31B. Every block we probed was loaded back from the encrypted store with no recompute (100 of 100, at depths from 0 to 50M tokens) on both models. Loading a block was 2.8x to 4.3x faster than recomputing it and used 8.8x to 12.3x less GPU energy, and GPU memory stayed flat over the whole 50M-token stream. Asked about facts planted millions of tokens earlier, the 12B model gave the right answer 82 times out of 100 and the 31B model 98 times out of 100. Neither model made up an answer. The limits are as follows. This is reuse of stored state, not a wider attention window: one block is loaded at a time, and how well a question is answered depends on the model. Writing the memory is a one-time cost, and the store takes terabytes of local NVMe disk. We describe the test protocol, which is built to resist common ways of gaming long-context benchmarks, and give a single-GPU reproduction that uses public software and a free licence for the package.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 23 of its 30 references that are in the map. First posted on arXiv 7 Oct 2026.

Builds on

23 of 30 references.

References · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  5. 2025Gemma 3 Technical Reportcited by 1,686
  6. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
  7. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  8. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  9. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  10. 2022Efficiently Scaling Transformer Inferencecited by 529
  11. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  12. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 242
11 more
  1. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 211
  2. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 199
  3. 2026Gemma 4 Technical Reportcited by 187
  4. 2023Time Travel in LLMs: Tracing Data Contamination in Large Language Modelscited by 175
  5. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 159
  6. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 119
  7. 2024Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Taskscited by 38
  8. 2024Do Large Language Models Need a Content Delivery Network?cited by 15
  9. 2026Smarter and Cheaper at Once: Byte-Exact KV-Cache Grafting Turns a Frozen Small Model into a Verified-Knowledge Flywheelcited by 2
  10. 2026A Frozen 12B Beats Frontier Models on Verified Work: 100% Accuracy, 0 Tokens, Bit-Exact, Forevercited by 2
  11. 2026Working Around the Compute Ceiling: Byte-Exact Memory in Galahad Makes LLM Reading a One-Time Cost LLM Reading a One-Time Costcited by 1

Built on by

Not cited yet.