mlsanityData to 7 Oct 2026

MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Engines

arXiv 2409.12959 ↗First posted 19 Sep 2024cs.CV cs.AI cs.CLcited by 98

Not on the map. Why?

Cited by 52 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The advent of Large Language Models (LLMs) has paved the way for AI search engines, e.g., SearchGPT, showcasing a new paradigm in human-internet interaction. However, most current AI search engines are limited to text-only settings, neglecting the multimodal user queries and the text-image interleaved nature of website information. Recently, Large Multimodal Models (LMMs) have made impressive strides. Yet, whether they can function as AI search engines remains under-explored, leaving the potential of LMMs in multimodal search an open question. To this end, we first design a delicate pipeline, MMSearch-Engine, to empower any LMMs with multimodal search capabilities. On top of this, we introduce MMSearch, a comprehensive evaluation benchmark to assess the multimodal search performance of LMMs. The curated dataset contains 300 manually collected instances spanning 14 subfields, which involves no overlap with the current LMMs' training data, ensuring the correct answer can only be obtained within searching. By using MMSearch-Engine, the LMMs are evaluated by performing three individual tasks (requery, rerank, and summarization), and one challenging end-to-end task with a complete searching process. We conduct extensive experiments on closed-source and open-source LMMs. Among all tested models, GPT-4o with MMSearch-Engine achieves the best results, which surpasses the commercial product, Perplexity Pro, in the end-to-end task, demonstrating the effectiveness of our proposed pipeline. We further present error analysis to unveil current LMMs still struggle to fully grasp the multimodal search tasks, and conduct ablation study to indicate the potential of scaling test-time computation for AI search engine. We hope MMSearch may provide unique insights to guide the future development of multimodal AI search engine. Project Page: https://mmsearch.github.io

Key facts. Cited by 98 papers in the map (data through 7 Oct 2026). Builds on 55 of its 72 references that are in the map. First posted on arXiv 19 Sep 2024.

Builds on

55 of 72 references.

References · 55

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  8. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  9. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  10. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  11. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  12. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
38 more
  1. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  2. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  3. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  4. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  5. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  6. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  7. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  8. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  9. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  10. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  11. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  12. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  13. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  14. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 450
  15. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
  16. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  17. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 307
  18. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 285
  19. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  20. 2024Chain of Thought Empowers Transformers to Solve Inherently Serial Problemscited by 256
  21. 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 212
  22. 2023Query Rewriting for Retrieval-Augmented Large Language Modelscited by 211
  23. 2023G-LLaVA: Solving Geometric Problem with Multi-Modal Large Language Modelcited by 208
  24. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  25. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 192
  26. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understandingcited by 178
  27. 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 177
  28. 2024RQ-RAG: Learning to Refine Queries for Retrieval Augmented Generationcited by 144
  29. 2023ImageBind-LLM: Multi-modality Instruction Tuningcited by 142
  30. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 130
  31. 2023VideoLLM: Modeling Video Sequence with Large Language Modelscited by 116
  32. 2024Large Multimodal Agents: A Surveycited by 99
  33. 2021WebSRC: A Dataset for Web-Based Structural Reading Comprehensioncited by 98
  34. 2024VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?cited by 98
  35. 2023LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understandingcited by 95
  36. 2024CRAG -- Comprehensive RAG Benchmarkcited by 94
  37. 2024Reducing hallucination in structured outputs via Retrieval-Augmented Generationcited by 92
  38. 2024MoVA: Adapting Mixture of Vision Experts to Multimodal Contextcited by 82

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 34; 2026: 60.

98 papers.

Sort:

By area:

  1. 2025T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoTcited by 162
  2. 2025Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Stepcited by 130
  3. 2025MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiencycited by 123
  4. 2025MMSearch-R1: Incentivizing LMMs to Searchcited by 120
  5. 2025DeepEyesV2: Toward Agentic Multimodal Modelcited by 119
  6. 2025WebWatcher: Breaking New Frontier of Vision-Language Deep Research Agentcited by 114
  7. 2025Echo-4o: Harnessing the Power of GPT-4o Synthetic Images for Improved Image Generationcited by 95
  8. 2025VRAG-RL: Empower Vision-Perception-Based RAG for Visually Rich Information Understanding via Iterative Reasoning with Reinforcement Learningcited by 68
  9. 2025WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratchcited by 66
  10. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  11. 2025MINT-CoT: Enabling Interleaved Visual Tokens in Mathematical Chain-of-Thought Reasoningcited by 45
  12. 2026REDSearcher: A Scalable and Cost-Efficient Framework for Long-Horizon Search Agentscited by 43
38 more
  1. 2025DeepMMSearch-R1: Empowering Multimodal LLMs in Multimodal Web Searchcited by 41
  2. 2025MM-BrowseComp: A Comprehensive Benchmark for Multimodal Browsing Agentscited by 39
  3. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  4. 2025MMSearch-Plus: Benchmarking Provenance-Aware Search for Multimodal Browsing Agentscited by 35
  5. 2024MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Modelscited by 32
  6. 2024PixWizard: Versatile Image-to-Image Visual Assistant with Open-Language Instructionscited by 31
  7. 2025SenseNova-MARS: Empowering Multimodal Agentic Reasoning and Search via Reinforcement Learningcited by 31
  8. 2026GLM-5V-Turbo: Toward a Native Foundation Model for Multimodal Agentscited by 30
  9. 2025Skywork-R1V4: Toward Agentic Multimodal Intelligence through Interleaved Thinking with Images and DeepResearchcited by 25
  10. 2025SciVerse: Unveiling the Knowledge Comprehension and Visual Reasoning of LMMs on Multi-modal Scientific Problemscited by 22
  11. 2025A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applicationscited by 21
  12. 2026MMDeepResearch-Bench: A Benchmark for Multimodal Deep Research Agentscited by 21
  13. 2025A Survey on Agentic Multimodal Large Language Modelscited by 19
  14. 2026MM-DeepResearch: A Simple and Effective Multimodal Agentic Search Baselinecited by 18
  15. 2026Watching, Reasoning, and Searching: A Video Deep Research Benchmark on Open Web for Agentic Video Reasoningcited by 14
  16. 2026LensWalk: Agentic Video Understanding by Planning How You See in Videoscited by 14
  17. 2025DraCo: Draft as CoT for Text-to-Image Preview and Rare Concept Generationcited by 13
  18. 2025MRMR: A Realistic and Expert-Level Multidisciplinary Benchmark for Reasoning-Intensive Multimodal Retrievalcited by 12
  19. 2025CrossLMM: Decoupling Long Video Sequences from LMMs via Dual Cross-Attention Mechanismscited by 11
  20. 2025AutoTool: Dynamic Tool Selection and Integration for Agentic Reasoningcited by 11
  21. 2026MC-Search: Evaluating and Enhancing Multimodal Agentic Search with Structured Long Reasoning Chainscited by 11
  22. 2026Who&When Pro: Can LLMs Really Attribute Failures in AI Agents?cited by 11
  23. 2025BLINK-Twice: You see, but do you observe? A Reasoning Benchmark on Visual Perceptioncited by 9
  24. 2025Video-Browser: Towards Agentic Open-web Video Browsingcited by 9
  25. 2026Towards Long-horizon Agentic Multimodal Searchcited by 9
  26. 2025PiSA: A Self-Augmented Data Engine and Training Strategy for 3D Understanding with Large Modelscited by 8
  27. 2026VisBrowse-Bench: Benchmarking Visual-Native Search for Multimodal Browsing Agentscited by 8
  28. 2026HyperEyes: Dual-Grained Efficiency-Aware Reinforcement Learning for Parallel Multimodal Search Agentscited by 8
  29. 2026MTA-Agent: An Open Recipe for Multimodal Deep Search Agentscited by 7
  30. 2025MindWatcher: Toward Smarter Multimodal Tool-Integrated Reasoningcited by 6
  31. 2026VSearcher: Long-Horizon Multimodal Search Agent via Reinforcement Learningcited by 6
  32. 2026Memory Intelligence Agentcited by 6
  33. 2026Towards On-Policy Data Evolution for Visual-Native Multimodal Deep Search Agentscited by 6
  34. 2024An Agent Framework for Real-Time Financial Information Searching with Large Language Modelscited by 5
  35. 2026MiroEval: Benchmarking Multimodal Deep Research Agents in Process and Outcomecited by 5
  36. 2026Visual-Seeker: Towards Visual-Native Multimodal Agentic Search via Active Visual Reasoningcited by 5
  37. 2026VideoSearcher: Empowering Video Deep Research with Multi-Tool Agentic Reasoning via Reinforcement Learningcited by 5
  38. 2025BioMedSearch: A Multi-Source Biomedical Retrieval Framework Based on LLMscited by 4

50 most cited of 98 All 98 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Evaluating Cost-Accuracy Trade-offs in Multimodal Search Relevance Judgements
  2. 2024CUE-M: Contextual Understanding and Enhanced Search with Multimodal Large Language Model
  3. 2024MM-Embed: Universal Multimodal Retrieval with Multimodal LLMs
  4. 2024Enhancing Cloud-Based Large Language Model Processing with Elasticsearch and Transformer Models
  5. 2023MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteria
  6. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Models
  7. 2024Retrieve, Annotate, Evaluate, Repeat: Leveraging Multimodal LLMs for Large-Scale Product Retrieval Evaluation
  8. 2026Vision-DeepResearch: Incentivizing DeepResearch Capability in Multimodal Large Language Models