# MM-ContextFold: Context Folding for Multimodal Agentic Retrieval

arXiv: https://arxiv.org/abs/2609.23121 · HTML: https://mlsanity.com/paper/2609.23121/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use.
- Builds on 21 of its 43 references that are in the map.
- First posted on arXiv 19 Sep 2026.

## Placed in

- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 21 of 21)

- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [A-MEM: Agentic Memory for LLM Agents](https://mlsanity.com/paper/2502.12110/) (2025)
- [Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory](https://mlsanity.com/paper/2504.19413/) (2025)
- [DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environments](https://mlsanity.com/paper/2504.03160/) (2025)
- [MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents](https://mlsanity.com/paper/2506.15841/) (2025)
- [WebDancer: Towards Autonomous Information Seeking Agency](https://mlsanity.com/paper/2505.22648/) (2025)
- [WebWalker: Benchmarking LLMs in Web Traversal](https://mlsanity.com/paper/2501.07572/) (2025)
- [ReSum: Unlocking Long-Horizon Search Intelligence via Context Summarization](https://mlsanity.com/paper/2509.13313/) (2025)
- [WebResearcher: Unleashing unbounded reasoning capability in Long-Horizon Agents](https://mlsanity.com/paper/2509.13309/) (2025)
- [REDSearcher: A Scalable and Cost-Efficient Framework for Long-Horizon Search Agents](https://mlsanity.com/paper/2602.14234/) (2026)
- [MMSearch-Plus: Benchmarking Provenance-Aware Search for Multimodal Browsing Agents](https://mlsanity.com/paper/2508.21475/) (2025)
- [Mitigating Visual Forgetting via Take-along Visual Conditioning for Multi-modal Long CoT Reasoning](https://mlsanity.com/paper/2503.13360/) (2025)
- [Seeking and Updating with Live Visual Knowledge](https://mlsanity.com/paper/2504.05288/) (2025)
- [Multimodal DeepResearcher: Generating Text-Chart Interleaved Reports From Scratch with Agentic Framework](https://mlsanity.com/paper/2506.02454/) (2025)
- [Rethinking Visual Token Reduction in LVLMs Under Cross-Modal Misalignment](https://mlsanity.com/paper/2506.22283/) (2025)
- [RedundancyLens: Revealing and Exploiting Visual Token Processing Redundancy for Efficient Decoder-Only MLLMs](https://mlsanity.com/paper/2501.19036/) (2025)
- [MindWatcher: Toward Smarter Multimodal Tool-Integrated Reasoning](https://mlsanity.com/paper/2512.23412/) (2025)
- [DeepImageSearch: Benchmarking Multimodal Agents for Context-Aware Image Retrieval in Visual Histories](https://mlsanity.com/paper/2602.10809/) (2026)
- [ProReason: Multi-Modal Proactive Reasoning with Decoupled Eyesight and Wisdom](https://mlsanity.com/paper/2410.14138/) (2024)
- [VimRAG: Navigating Massive Visual Context in Retrieval-Augmented Generation via Multimodal Memory Graph](https://mlsanity.com/paper/2602.12735/) (2026)
- [MLDocRAG: Multimodal Long-Context Document Retrieval Augmented Generation](https://mlsanity.com/paper/2602.10271/) (2026)

## Abstract (arXiv)

Multimodal Agentic Retrieval (MAR) requires agents to solve complex information-seeking tasks by iteratively invoking external tools. Typical frameworks such as ReAct maintain raw multimodal inputs and the accumulating interaction history in a single, ever-growing context, leading to the context explosion problem. While existing methods alleviate this issue by compressing redundant text, effective strategies for managing token-intensive visual content remain largely underexplored. To address this gap, we first conduct a systematic empirical study of approximately 10,000 trajectories. The results show that as visual cues are progressively extracted through external tools and textualized into the context, raw images become increasingly redundant. Continued image retention is associated with higher output entropy and can even degrade task accuracy. Motivated by these findings, we propose MM-ContextFold, a training-free framework that loads raw images only when needed. It maintains a persistent, text-only main context for high-level planning and spawns ephemeral branch contexts for image-dependent subtasks. Within each branch, the agent loads the relevant images, completes the subtask, and folds the result back into the main context as a concise textual summary; the images and branch trace are then discarded. Experiments on seven MAR benchmarks across five backbone models show that MM-ContextFold improves average accuracy by 6.3 percentage points over ReAct while reducing the working context length by 27.5%.