# Test-Time Memory Construction

arXiv: https://arxiv.org/abs/2610.13348 · HTML: https://mlsanity.com/paper/2610.13348/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 33 of its 38 references that are in the map.
- First posted on arXiv 9 Oct 2026.

## Builds on (most cited first, 30 of 33)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [From Local to Global: A Graph RAG Approach to Query-Focused Summarization](https://mlsanity.com/paper/2404.16130/) (2024)
- [MemGPT: Towards LLMs as Operating Systems](https://mlsanity.com/paper/2310.08560/) (2023)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation](https://mlsanity.com/paper/2402.03216/) (2024)
- [Evaluating Very Long-Term Conversational Memory of LLM Agents](https://mlsanity.com/paper/2402.17753/) (2024)
- [Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory](https://mlsanity.com/paper/2504.19413/) (2025)
- [LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory](https://mlsanity.com/paper/2410.10813/) (2024)
- [A Survey on the Memory Mechanism of Large Language Model based Agents](https://mlsanity.com/paper/2404.13501/) (2024)
- [MemoryBank: Enhancing Large Language Models with Long-Term Memory](https://mlsanity.com/paper/2305.10250/) (2023)
- [Search-o1: Agentic Search-Enhanced Large Reasoning Models](https://mlsanity.com/paper/2501.05366/) (2025)
- [Memory in the Age of AI Agents](https://mlsanity.com/paper/2512.13564/) (2025)
- [Token-Budget-Aware LLM Reasoning](https://mlsanity.com/paper/2412.18547/) (2024)
- [LightMem: Lightweight and Efficient Memory-Augmented Generation](https://mlsanity.com/paper/2510.18866/) (2025)
- [Memory OS of AI Agent](https://mlsanity.com/paper/2506.06326/) (2025)
- [In Prospect and Retrospect: Reflective Memory Management for Long-term Personalized Dialogue Agents](https://mlsanity.com/paper/2503.08026/) (2025)
- [How Well do LLMs Compress Their Own Chain-of-Thought? A Token Complexity Approach](https://mlsanity.com/paper/2503.01141/) (2025)
- [EverMemOS: A Self-Organizing Memory Operating System for Structured Long-Horizon Reasoning](https://mlsanity.com/paper/2601.02163/) (2026)
- [TiMem: Temporal-Hierarchical Memory Consolidation for Long-Horizon Conversational Agents](https://mlsanity.com/paper/2601.02845/) (2026)
- [Beyond RAG for Agent Memory: Retrieval by Decoupling and Aggregation](https://mlsanity.com/paper/2602.02007/) (2026)
- [HiMem: Hierarchical Long-Term Memory for LLM Long-Horizon Agents](https://mlsanity.com/paper/2601.06377/) (2026)
- [AttnTrace: Contextual Attribution of Prompt Injection and Knowledge Corruption](https://mlsanity.com/paper/2508.03793/) (2025)
- [MetaMem: Evolving Meta-Memory for Knowledge Utilization through Self-Reflective Symbolic Optimization](https://mlsanity.com/paper/2602.11182/) (2026)
- [LazyMem: Retrieve Broadly, Construct Selectively for Efficient Long-Term Agent Memory](https://mlsanity.com/paper/2607.22690/) (2026)
- [Finding What Matters: Anchoring Context Knowledge with Evolving Indices for Iterative Retrieval](https://mlsanity.com/paper/2601.16462/) (2026)
- [LISRec: Modeling User Preferences with Learned Item Shortcuts for Sequential Recommendation](https://mlsanity.com/paper/2505.22130/) (2025)
- [SEEK: Steering LLM Reasoning for RAG via Internal Reasoning Sketches](https://mlsanity.com/paper/2601.09402/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) struggle to answer questions over long interaction histories, where relevant evidence may be scattered across distant sessions. Existing memory systems typically construct and maintain sophisticated memory representations offline, before user queries are observed, incurring substantial overhead and limiting their ability to adapt memory construction to query-specific information needs. In this paper, we propose T2Mem, a Test-Time Memory construction framework that defers memory construction to inference and progressively builds memory along the reasoning process. Specifically, T2Mem extracts lightweight episode-level metadata offline to capture key events and temporal associations as retrieval cues, while retaining links to the original interactions to facilitate the reconstruction of dependencies across episodes. At test time, the LLM adaptively retrieves and organizes metadata and raw interactions, progressively constructing query-specific memory as reasoning unfolds. Experiments on LoCoMo and LongMemEval show the effectiveness of T2Mem by consistently outperforming strong baselines across different backbone LLMs while substantially reducing the overhead of memory construction. Further analysis demonstrates that deferring memory construction to inference enables the memory construction process to dynamically adapt to the information needs of individual queries. All code is available at https://github.com/OpenBMB/T2Mem.