# REMORY: Learning Residual Memory for Context Compaction

arXiv: https://arxiv.org/abs/2610.11287 · HTML: https://mlsanity.com/paper/2610.11287/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 37 references that are in the map.
- First posted on arXiv 8 Oct 2026.

## Builds on (most cited first, 11 of 11)

- [MemGPT: Towards LLMs as Operating Systems](https://mlsanity.com/paper/2310.08560/) (2023)
- [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://mlsanity.com/paper/2504.12516/) (2025)
- [Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces](https://mlsanity.com/paper/2601.11868/) (2026)
- [ReSum: Unlocking Long-Horizon Search Intelligence via Context Summarization](https://mlsanity.com/paper/2509.13313/) (2025)
- [DFlash: Block Diffusion for Flash Speculative Decoding](https://mlsanity.com/paper/2602.06036/) (2026)
- [AgentFold: Long-Horizon Web Agents with Proactive Context Management](https://mlsanity.com/paper/2510.24699/) (2025)
- [MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methods](https://mlsanity.com/paper/2601.21821/) (2026)
- [AutomationBench](https://mlsanity.com/paper/2604.18934/) (2026)
- [JobBench: Aligning Agent Work With Human Will](https://mlsanity.com/paper/2605.26329/) (2026)
- [WebAnchor: Anchoring Agent Planning to Stabilize Long-Horizon Web Reasoning](https://mlsanity.com/paper/2601.03164/) (2026)
- [End-to-End Context Compression at Scale](https://mlsanity.com/paper/2606.09659/) (2026)

## Abstract (arXiv)

Long-horizon agents compact their history to continue within a finite context window, but a textual summary alone may not support every subsequent decision. We introduce REMORY, a neural memory network that supplements the summary with a bounded sequence of soft memory tokens. Given the history and summary, the network learns to generate tokens that help a frozen LLM approximate the continuation it would produce with the full history. The tokens are conditioned on the summary and appended after it, forming an analogue of a residual connection along the sequence dimension. On SummHay, REMORY improves source attribution at nearly unchanged insight coverage and approaches the full-context joint score using only 5.2% of the input positions. Across long-horizon agent benchmarks, Qwen3.8-27B and GLM-5.3-Flash show consistent gains with residual memory. Both models also exhibit substantially fewer repeated tool outputs and tool errors on BrowseComp and Terminal-Bench 2.1.