# The Missing Memory Hierarchy: Demand Paging for LLM Context Windows

arXiv: https://arxiv.org/abs/2603.09023 · HTML: https://mlsanity.com/paper/2603.09023/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 20 references that are in the map.
- First posted on arXiv 9 Mar 2026.

## Builds on (most cited first, 6 of 6)

- [MemGPT: Towards LLMs as Operating Systems](https://mlsanity.com/paper/2310.08560/) (2023)
- [LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression](https://mlsanity.com/paper/2310.06839/) (2023)
- [ACON: Optimizing Context Compression for Long-horizon LLM Agents](https://mlsanity.com/paper/2510.00615/) (2025)
- [SWE-Pruner: Self-Adaptive Context Pruning for Coding Agents](https://mlsanity.com/paper/2601.16746/) (2026)
- [Active Context Compression: Autonomous Memory Management in LLM Agents](https://mlsanity.com/paper/2601.07190/) (2026)
- [SideQuest: Model-Driven KV Cache Management for Long-Horizon Agentic Reasoning](https://mlsanity.com/paper/2602.22603/) (2026)

## Built on by (most cited first, 8 of 8)

- [TokenPilot: Cache-Efficient Context Management for LLM Agents](https://mlsanity.com/paper/2606.17016/) (2026)
- [LLM Agents Are Latent Context Managers: Eliciting Self-Managed Context via State Proprioception](https://mlsanity.com/paper/2606.30005/) (2026)
- [The Working Set of a Coding Agent: Coherence Debt in Repository-Scale Tasks](https://mlsanity.com/paper/2608.16630/) (2026)
- [Cooperative Memory Paging with Keyword Bookmarks for Long-Horizon LLM Conversations](https://mlsanity.com/paper/2604.12376/) (2026)
- [Blast Radius](https://mlsanity.com/paper/2608.07440/) (2026)
- [No=esis: Deterministic-First Retrieval with Two-Tier Context Hydration for Factuality-Critical Queries on Small Local Models](https://mlsanity.com/paper/2609.07663/) (2026)
- [What Eviction Destroys: A Restore-Counterfactual Audit of Forgetting in Agent Memory](https://mlsanity.com/paper/2609.08279/) (2026)
- [CADOC: Cache-Aware Dynamic Object Context for Long-Horizon Agents](https://mlsanity.com/paper/2609.37012/) (2026)

## Abstract (arXiv)

The context window of a large language model is not memory. It is L1 cache: a small, fast, expensive resource that the field treats as the entire memory system. There is no L2, no virtual memory, no paging. Every tool definition, every system prompt, and every stale tool result occupies context for the lifetime of the session. The result is measurable: across 857 production sessions and 4.45 million effective input tokens, 21.8% is structural waste. We present Pichay, a demand paging system for LLM context windows. Implemented as a transparent proxy between client and inference API, Pichay interposes on the message stream to evict stale content, detect page faults when the model re-requests evicted material, and pin working-set pages identified by fault history. In offline replay across 1.4 million simulated evictions, the fault rate is 0.0254%. In live production deployment over 681turns, the system reduces context consumption by up to 93% (5,038KB to 339KB); under extreme sustained pressure, the system remains operational but exhibits the expected thrashing pathology, with repeated fault-in of evicted content. The key observation is that the problems the field faces, such as context limits, attention degradation, cost scaling, lost state across sessions, are virtual memory problems wearing different clothes. The solutions exist: working set theory (Denning, 1968), demand paging, fault-driven replacement policies, and memory hierarchies with multiple eviction-managed levels. We describe the architecture of a full memory hierarchy for LLM systems (L1 through persistent storage), report on the first three levels deployed in production use (L1 eviction, L2 fault-driven pinning, L3 model-initiated conversation compaction), and identify cross-session memory as the remaining frontier.