POINTS-Seeker: An Open Recipe for Multimodal Search Agents with Visual Memory Management
On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG
Abstract (arXiv)
Large Multimodal Models (LMMs) excel at visual perception but struggle with real-time, knowledge-intensive queries due to their reliance on static parametric knowledge. While multimodal search agents offer a promising solution, developing them from vanilla LMMs presents two major challenges: the lack of open recipes to cultivate search agency from scratch, and the severe context explosion (attention dilution) that occurs during long-horizon multi-turn investigations. In this paper, we address both challenges by developing a native multimodal search agent. First, we introduce an open training recipe featuring Agentic Seeding, a formative training stage that bootstraps tool-use and planning abilities directly from a non-agentic foundation. Second, to overcome the context bottleneck, we propose V-Fold, an adaptive memory management mechanism. V-Fold retains recent interactions as high-fidelity text while folding stale historical context into the visual space via rendering, exploiting the model's cross-modal alignment to preserve raw evidence without text token redundancy. Combining these innovations, we present POINTS-Seeker-8B, which achieves state-of-the-art performance among models of comparable scale across six multimodal search benchmarks.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 33 of its 52 references that are in the map. First posted on arXiv 15 Apr 2026.
Builds on
33 of 52 references.
Closest ancestors · 7
Same direction, newest first
- 2025DeepMMSearch-R1: Empowering Multimodal LLMs in Multimodal Web Search14 Oct
- 2025POINTS-Reader: Distillation-Free Adaptation of Vision-Language Models for Document Conversion1 Sep
- 2025MMSearch-R1: Incentivizing LMMs to Search25 Jun
- 2024LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant2 Dec
- 2024MLLM Is a Strong Reranker: Advancing Multimodal Retrieval-augmented Generation via Knowledge-enhanced Reranking and Noise-injected Training31 Jul
- 2024EchoSight: Advancing Visual-Language Models with Wiki Knowledge17 Jul
- 2022REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory10 Dec
Other references · 26
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
- 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
- 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
- 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
- 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
- 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
- 2024OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generationcited by 332
- 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 237
- 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Textcited by 216
- 2023Encyclopedic VQA: Visual questions about detailed properties of fine-grained categoriescited by 105
14 more
- 2025VRAG-RL: Empower Vision-Perception-Based RAG for Visually Rich Information Understanding via Iterative Reasoning with Reinforcement Learningcited by 68
- 20251.4 Million Open-Source Distilled Reasoning Dataset to Empower Large Language Model Trainingcited by 49
- 2026REDSearcher: A Scalable and Cost-Efficient Framework for Long-Horizon Search Agentscited by 43
- 2025MMSearch-Plus: Benchmarking Provenance-Aware Search for Multimodal Browsing Agentscited by 35
- 2025VisualWebInstruct: Scaling up Multimodal Instruction Data through Web Searchcited by 34
- 2025Bee: A High-Quality Corpus and Full-Stack Suite to Unlock Advanced Fully Open MLLMscited by 27
- 2025Seeking and Updating with Live Visual Knowledgecited by 23
- 2026AgentOCR: Reimagining Agent History via Optical Self-Compressioncited by 22
- 2026MM-DeepResearch: A Simple and Effective Multimodal Agentic Search Baselinecited by 18
- 2026MemOCR: Layout-Aware Visual Memory for Efficient Long-Horizon Reasoningcited by 15
- 2025Open Data Synthesis For Deep Researchcited by 14
- 2026Towards Long-horizon Agentic Multimodal Searchcited by 9
- 2024MLAN: Language-Based Instruction Tuning Preserves and Transfers Knowledge in Multimodal Language Modelscited by 5
- 2026Weaver: End-to-End Agentic System Training for Video Interleaved Reasoningcited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025InSight-o3: Empowering Multimodal Foundation Models with Generalized Visual Search
- 2026DeepVoyager-VL: Incentivizing Vision-in-the-Loop Search for Long-Horizon Multimodal Agents
- 2026VimRAG: Navigating Massive Visual Context in Retrieval-Augmented Generation via Multimodal Memory Graph
- 2026DeepImageSearch: Benchmarking Multimodal Agents for Context-Aware Image Retrieval in Visual Histories
- 2026LiteSearch-VL: Small Multimodal Search Agents via Trajectory Distillation and Synthetic Step-DPO
- 2025SenseNova-MARS: Empowering Multimodal Agentic Reasoning and Search via Reinforcement Learning
- 2026ProMMSearchAgent: A Generalizable Multimodal Search Agent Trained with Process-Oriented Rewards
- 2026OpenSearch-VL: An Open Recipe for Frontier Multimodal Search Agents