mlsanityData to 13 Oct 2026

GME: Improving Universal Multimodal Retrieval by Multimodal LLMs

arXiv 2412.16855 ↗First posted 22 Dec 2024cs.CL cs.IRcited by 167

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

Universal Multimodal Retrieval (UMR) aims to enable search across various modalities using a unified model, where queries and candidates can consist of pure text, images, or a combination of both. Previous work has attempted to adopt multimodal large language models (MLLMs) to realize UMR using only text data. However, our preliminary experiments demonstrate that more diverse multimodal training data can further unlock the potential of MLLMs. Despite its effectiveness, the existing multimodal training data is highly imbalanced in terms of modality, which motivates us to develop a training data synthesis pipeline and construct a large-scale, high-quality fused-modal training dataset. Based on the synthetic training data, we develop the General Multimodal Embedder (GME), an MLLM-based dense retriever designed for UMR. Furthermore, we construct a comprehensive UMR Benchmark (UMRB) to evaluate the effectiveness of our approach. Experimental results show that our method achieves state-of-the-art performance among existing UMR methods. Last, we provide in-depth analyses of model scaling and training strategies, and perform ablation studies on both the model and synthetic data.

Key facts. Cited by 167 papers in the map (data through 13 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 63 of its 80 references that are in the map. First posted on arXiv 22 Dec 2024.

Builds on

63 of 80 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks7 Oct
  2. 2024E5-V: Universal Embeddings with Multimodal Large Language Models17 Jul
  3. 2024ColPali: Efficient Document Retrieval with Vision Language Models27 Jun

Other references · 60

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  2. 2020Language Models are Few-Shot Learnerscited by 32,320
  3. 2023GPT-4 Technical Reportcited by 18,278
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,550
  5. 2023Visual Instruction Tuningcited by 7,827
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,712
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,314
  8. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,191
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,141
  10. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,133
  11. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,797
  12. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,716
38 more
  1. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,925
  2. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,170
  3. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,600
  4. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,368
  5. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,355
  6. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,295
  7. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,237
  8. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,229
  9. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,205
  10. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,116
  11. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,070
  12. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,041
  13. 2023A Survey on Multimodal Large Language Modelscited by 905
  14. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 847
  15. 2016Training Deep Nets with Sublinear Memory Costcited by 840
  16. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 714
  17. 2020Contrastive Learning with Hard Negative Samplescited by 619
  18. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 562
  19. 2020Fact or Fiction: Verifying Scientific Claimscited by 552
  20. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 467
  21. 2021InfographicVQAcited by 431
  22. 2020SPECTER: Document-level Representation Learning using Citation-informed Transformerscited by 429
  23. 2023DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Datacited by 327
  24. 2023Improving Text Embeddings with Large Language Modelscited by 297
  25. 2021Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Modelscited by 249
  26. 2022Promptagator: Few-shot Dense Retrieval From 8 Examplescited by 233
  27. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 205
  28. 2020CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claimscited by 195
  29. 2024Building and better understanding vision-language models: insights and future directionscited by 172
  30. 2023UniIR: Training and Benchmarking Universal Multimodal Information Retrieverscited by 166
  31. 2024Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Modelscited by 139
  32. 2022Dense Text Retrieval based on Pretrained Language Models: A Surveycited by 136
  33. 2017Automatic Spatially-aware Fashion Concept Discoverycited by 133
  34. 2020Visual News: Benchmark and Challenges in News Image Captioningcited by 121
  35. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 118
  36. 2023Open-domain Visual Entity Recognition: Towards Recognizing Millions of Wikipedia Entitiescited by 112
  37. 2021WebQA: Multihop and Multimodal QAcited by 109
  38. 2024Unifying Multimodal Retrieval via Document Screenshot Embeddingcited by 108

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 65; 2026: 102.

167 papers.

Sort:

By area:

  1. 2025VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documentscited by 113
  2. 2025OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generationcited by 73
  3. 2025jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrievalcited by 72
  4. 2026Mem-Gallery: Benchmarking Multimodal Long-Term Conversational Memory for MLLM Agentscited by 54
  5. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  6. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  7. 2025Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrievalcited by 26
  8. 2026Embed-RL: Reinforcement Learning for Reasoning-Driven Multimodal Embeddingscited by 24
  9. 2025MAGREF: Masked Guidance for Any-Reference Video Generation with Subject Disentanglementcited by 23
  10. 2025MM-R5: MultiModal Reasoning-Enhanced ReRanker via Reinforcement Learning for Document Retrievalcited by 21
  11. 2025UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAGcited by 21
  12. 2025Scaling Language-Centric Omnimodal Representation Learningcited by 21
38 more
  1. 2025DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruningcited by 19
  2. 2025WavRAG: Audio-Integrated Retrieval Augmented Generation for Spoken Dialogue Modelscited by 18
  3. 2025FinRAGBench-V: A Benchmark for Multimodal RAG with Visual Citation in the Financial Domaincited by 16
  4. 2025BookRAG: A Hierarchical Structure-aware Index-based Approach for Retrieval-Augmented Generation on Complex Documentscited by 16
  5. 2025LoVR: A Benchmark for Long Video Retrieval in Multimodal Contextscited by 14
  6. 2025VisR-Bench: An Empirical Study on Visual Retrieval-Augmented Generation for Multilingual Long Document Understandingcited by 13
  7. 2026e5-omni: Explicit Cross-modal Alignment for Omni-modal Embeddingscited by 12
  8. 2025Robust Adaptation of Large Multimodal Models for Retrieval Augmented Hateful Meme Detectioncited by 11
  9. 2026PLUME: Latent Reasoning Based Universal Multimodal Embeddingcited by 11
  10. 2025U-MARVEL: Unveiling Key Factors for Universal Multimodal Retrieval via Embedding Learning with MLLMscited by 10
  11. 2025UniECS: Unified Multimodal E-Commerce Search Framework with Gated Cross-modal Fusioncited by 10
  12. 2025SAIL-Embedding Technical Report: Omni-modal Embedding Foundation Modelcited by 10
  13. 2025MIRACL-VISION: A Large, multilingual, visual document retrieval benchmarkcited by 9
  14. 2025Identity-GRPO: Optimizing Multi-Human Identity-preserving Video Generation via Reinforcement Learningcited by 9
  15. 2026Sculpting the Vector Space: Towards Efficient Multi-Vector Visual Document Retrieval via Prune-then-Merge Frameworkcited by 9
  16. 2025Any Information Is Just Worth One Single Screenshot: Unifying Search With Visualized Information Retrievalcited by 8
  17. 2025IDMR: Towards Instance-Driven Precise Visual Correspondence in Multimodal Retrievalcited by 8
  18. 2025One Pic is All it Takes: Poisoning Visual Document Retrieval Augmented Generation with a Single Imagecited by 8
  19. 2025Identity-Preserving Text-to-Video Generation via Training-Free Prompt, Image, and Guidance Enhancementcited by 8
  20. 2025ReMatch: Boosting Representation through Matching for Multimodal Retrievalcited by 8
  21. 2026Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrievalcited by 8
  22. 2025MOON: Generative MLLM-based Multimodal Representation Learning for E-commerce Product Understandingcited by 7
  23. 2025WAVE: Learning Unified & Versatile Audio-Visual Embeddings with Multimodal LLMcited by 7
  24. 2025FreeRet: MLLMs as Training-Free Retrieverscited by 7
  25. 2025Generalized Contrastive Learning for Universal Multimodal Retrievalcited by 7
  26. 2025MegaRAG: Multimodal Knowledge Graph-Based Retrieval Augmented Generationcited by 7
  27. 2026TRACE: Task-Adaptive Reasoning and Representation Learning for Universal Multimodal Retrievalcited by 7
  28. 2026ARGUS: Stacked Multi-View Identity Mosaic Injection for Subject-Preserving Video Generationcited by 7
  29. 2025Joint Fusion and Encoding: Advancing Multimodal Retrieval from the Ground Upcited by 6
  30. 2025A Multi-Granularity Retrieval Framework for Visually-Rich Documentscited by 6
  31. 2026CausalEmbed: Auto-Regressive Multi-Vector Generation in Latent Space for Visual Document Embeddingcited by 6
  32. 2026DeepImageSearch: Benchmarking Multimodal Agents for Context-Aware Image Retrieval in Visual Historiescited by 6
  33. 2026LMEB: Long-horizon Memory Embedding Benchmarkcited by 6
  34. 2026MMEmb-R1: Reasoning-Enhanced Multimodal Embedding with Pair-Aware Selection and Adaptive Controlcited by 6
  35. 2026Visual Late Chunking: An Empirical Study of Contextual Chunking for Efficient Visual Document Retrievalcited by 6
  36. 2026H2HMem: A Multimodal Memory Benchmark for Agents in Human-Human Interactionscited by 6
  37. 2025Are We on the Right Way for Assessing Document Retrieval-Augmented Generation?cited by 5
  38. 2025MOON2.0: Dynamic Modality-balanced Multimodal Representation Learning for E-commerce Product Understandingcited by 5

50 most cited of 167 All 167 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024MM-Embed: Universal Multimodal Retrieval with Multimodal LLMs
  2. 2026RETLLM: Training and Data-Free MLLMs for Multimodal Information Retrieval
  3. 2024MegaPairs: Massive Data Synthesis For Universal Multimodal Retrieval
  4. 2026Skim and Skip: Hierarchical Adaptive Inference for Efficient Multimodal Retrieval
  5. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Survey
  6. 2025PUMA: Layer-Pruned Language Model for Efficient Unified Multimodal Retrieval with Modality-Adaptive Learning
  7. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs
  8. 2024How Does the Textual Information Affect the Retrieval of Multimodal In-Context Learning?