# Memory in the Loop: In-Process Retrieval as Extended Working Memory for Language Agents

arXiv: https://arxiv.org/abs/2607.05690 · HTML: https://mlsanity.com/paper/2607.05690/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Agent memory systems, one of 16 directions in LLM agents and tool use.
- Builds on 27 of its 38 references that are in the map.
- First posted on arXiv 6 Jul 2026.

## Placed in

- LLM agents and tool use / [Agent memory systems](https://mlsanity.com/agents/agent-memory-systems/)

## Builds on (most cited first, 27 of 27)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [MemGPT: Towards LLMs as Operating Systems](https://mlsanity.com/paper/2310.08560/) (2023)
- [A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2405.06211/) (2024)
- [Cognitive Architectures for Language Agents](https://mlsanity.com/paper/2309.02427/) (2023)
- [Memory in the Age of AI Agents](https://mlsanity.com/paper/2512.13564/) (2025)
- [Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions](https://mlsanity.com/paper/2507.05257/) (2025)
- [From Human Memory to AI Memory: A Survey on Memory Mechanisms in the Era of LLMs](https://mlsanity.com/paper/2504.15965/) (2025)
- [MemBench: Towards More Comprehensive Evaluation on the Memory of LLM-based Agents](https://mlsanity.com/paper/2506.21605/) (2025)
- [Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering](https://mlsanity.com/paper/2604.08224/) (2026)
- [Memory for Autonomous LLM Agents:Mechanisms, Evaluation, and Emerging Frontiers](https://mlsanity.com/paper/2603.07670/) (2026)
- [Beyond RAG for Agent Memory: Retrieval by Decoupling and Aggregation](https://mlsanity.com/paper/2602.02007/) (2026)
- [Empowering Working Memory for Large Language Model Agents](https://mlsanity.com/paper/2312.17259/) (2023)
- [Win Fast or Lose Slow: Balancing Speed and Accuracy in Latency-Sensitive Decisions of LLMs](https://mlsanity.com/paper/2505.19481/) (2025)
- [Agent Memory: Characterization and System Implications of Stateful Long-Horizon Workloads](https://mlsanity.com/paper/2606.06448/) (2026)
- [Governed Memory: A Production Architecture for Multi-Agent Workflows](https://mlsanity.com/paper/2603.17787/) (2026)
- [Demystifying and Enhancing the Efficiency of Large Language Model Based Search Agents](https://mlsanity.com/paper/2505.12065/) (2025)
- [Ask Only When Needed: Proactive Retrieval from Memory and Skills for Experience-Driven Lifelong Agents](https://mlsanity.com/paper/2604.20572/) (2026)
- [Toward a New Science of AI as Cognitive Infrastructure](https://mlsanity.com/paper/2507.22893/) (2025)
- [AgentIR: A Workload-Adaptive Cascade Retrieval Substrate for Long-Term Conversational Memory](https://mlsanity.com/paper/2605.25092/) (2026)
- [Operationalising Extended Cognition: Formal Metrics for Corporate Knowledge and Legal Accountability](https://mlsanity.com/paper/2510.16193/) (2025)
- [AMV-L: Lifecycle-Managed Agent Memory for Tail-Latency Control in Long-Running LLM Systems](https://mlsanity.com/paper/2603.04443/) (2026)
- [The Library Theorem: How External Organization Governs Agentic Reasoning Capacity](https://mlsanity.com/paper/2603.21272/) (2026)
- [Entity-Collision: A Stratified Protocol for Attributing Retrieval Lift in Agent Memory](https://mlsanity.com/paper/2605.29630/) (2026)

## Abstract (arXiv)

Language agents run a loop - observe, reason, act - but the memory they reason over sits outside it: a store queried at most once per turn. We study the regime where memory moves inside the loop, read and written on every step. The obstacle has always been latency: networked stores answer in tens to hundreds of milliseconds, and in-loop retrieval can inflate end-to-end latency by up to 83x when retrieval is expensive. Prior work manages that cost rather than questioning it: serving-layer scheduling hides it, "memory-first" designs ration retrieval to once per turn. We argue latency is a property of where the store lives, not the in-loop pattern: an in-process store answers in ~100us, three orders of magnitude below the network regime, and at that speed the per-step tax collapses. By the extended-mind thesis's parity principle, a store fast enough to be constantly and directly available becomes extended working memory, not a tool the agent merely consults. The premise is causal: holding a fixed per-turn memory-latency budget and varying only the store's answer speed, redundant actions rise monotonically with latency - 0.0 of 12 at in-process speed, 7.2 of 12 at a 110ms cloud round trip (gpt-5-nano, gpt-5-mini; exact permutation p=0.0079). We demonstrate the regime end-to-end: across four GPT-5-class models under a bounded window, recall improves from 0/5 to 3.6-4.8/5 with in-loop memory, store ops at p50 80-165us - though an instructed restate-every-reply baseline also solves it perfectly, at a token cost that grows with the working set. The store never lost a fact in any run (244 of 244 writes kept); every miss traces to the agent's read policy, not the store. Our measurements also relocate the bottleneck: the dominant per-step cost is embedding (~200-400ms over the network); pairing the in-process store with a small local embedder returns the complete operation to a measured ~40us.