mlsanityData to 14 Oct 2026

MegaPairs: Massive Data Synthesis For Universal Multimodal Retrieval

arXiv 2412.14475 ↗First posted 19 Dec 2024cs.CV cs.CLcited by 84

On the map: Synthetic data and self-improvement for LLMs → Self-improving MLLMs and synthetic data

Abstract (arXiv)

Despite the rapidly growing demand for multimodal retrieval, progress in this field remains severely constrained by a lack of training data. In this paper, we introduce MegaPairs, a novel data synthesis method that leverages vision language models (VLMs) and open-domain images, together with a massive synthetic dataset generated from this method. Our empirical analysis shows that MegaPairs generates high-quality data, enabling the multimodal retriever to significantly outperform the baseline model trained on 70× more data from existing datasets. Moreover, since MegaPairs solely relies on general image corpora and open-source VLMs, it can be easily scaled up, enabling continuous improvements in retrieval performance. In this stage, we produced more than 26 million training instances and trained several models of varying sizes using this data. These new models achieve state-of-the-art zero-shot performance across 4 popular composed image retrieval (CIR) benchmarks and the highest overall performance on the 36 datasets provided by MMEB. They also demonstrate notable performance improvements with additional downstream fine-tuning. Our produced dataset, well-trained models, and data synthesis pipeline will be made publicly available to facilitate the future development of this field.

Key facts. Cited by 84 papers in the map (data through 14 Oct 2026). Placed in Self-improving MLLMs and synthetic data, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 37 of its 61 references that are in the map. First posted on arXiv 19 Dec 2024.

Builds on

37 of 61 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models30 Jan

Other references · 36

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  8. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  12. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
24 more
  1. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  2. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  3. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
  4. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  5. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 849
  6. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  7. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  8. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 333
  9. 2023Improving Text Embeddings with Large Language Modelscited by 303
  10. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 245
  11. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Taskscited by 228
  12. 2024MM-Embed: Universal Multimodal Retrieval with Multimodal LLMscited by 161
  13. 2024E5-V: Universal Embeddings with Multimodal Large Language Modelscited by 126
  14. 2022Retrieval-Augmented Multimodal Language Modelingcited by 125
  15. 2024MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructionscited by 119
  16. 2022Task-aware Retrieval with Instructionscited by 107
  17. 2023Vision-by-Language for Training-Free Compositional Image Retrievalcited by 103
  18. 2024What If We Recaption Billions of Web Images with LLaMA-3?cited by 87
  19. 2023CompoDiff: Versatile Composed Image Retrieval With Latent Diffusioncited by 85
  20. 2023A Multi-Task Embedder For Retrieval Augmented LLMscited by 64
  21. 2022Universal Vision-Language Dense Retrieval: Learning A Unified Representation Space for Multi-Modal Retrievalcited by 58
  22. 2024Imagine and Seek: Improving Composed Image Retrieval with an Imagined Proxycited by 18
  23. 2023Pretrain like Your Inference: Masked Tuning Improves Zero-Shot Composed Image Retrievalcited by 14
  24. 2024UniRAG: Universal Retrieval Augmentation for Large Vision Language Modelscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 39; 2026: 45.

84 papers.

Sort:

By area:

  1. 2026Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Rankingcited by 230
  2. 2025mmE5: Improving Multimodal Multilingual Embeddings via High-quality Synthetic Datacited by 49
  3. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  4. 2025Think Then Embed: Generative Context Improves Multimodal Embeddingcited by 41
  5. 2025MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interactioncited by 39
  6. 2025UME-R1: Exploring Reasoning-Driven Generative Multimodal Embeddingscited by 38
  7. 2025Video-XL-2: Towards Very Long-Video Understanding Through Task-Aware KV Sparsificationcited by 37
  8. 2025MemVerse: Multimodal Memory for Lifelong Learning Agentscited by 35
  9. 2025RzenEmbed: Towards Comprehensive Multimodal Retrievalcited by 29
  10. 2025Breaking the Batch Barrier (B3) of Contrastive Learning via Smart Batch Miningcited by 28
  11. 2025Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrievalcited by 26
  12. 2026Embed-RL: Reinforcement Learning for Reasoning-Driven Multimodal Embeddingscited by 24
38 more
  1. 2025MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddingscited by 20
  2. 2025Visual-RAG: Benchmarking Text-to-Image Retrieval Augmented Generation for Visual Knowledge Intensive Queriescited by 16
  3. 2025MomentSeeker: A Task-Oriented Benchmark For Long-Video Moment Retrievalcited by 14
  4. 2025Retrv-R1: A Reasoning-Driven MLLM Framework for Universal and Efficient Multimodal Retrievalcited by 12
  5. 2026MC-Search: Evaluating and Enhancing Multimodal Agentic Search with Structured Long Reasoning Chainscited by 12
  6. 2026PLUME: Latent Reasoning Based Universal Multimodal Embeddingcited by 11
  7. 2026Sculpting the Vector Space: Towards Efficient Multi-Vector Visual Document Retrieval via Prune-then-Merge Frameworkcited by 10
  8. 2025Towards Universal Video Retrieval: Generalizing Video Embedding via Synthesized Multimodal Pyramid Curriculumcited by 9
  9. 2025Reasoning Guided Embeddings: Leveraging MLLM Reasoning for Improved Multimodal Retrievalcited by 9
  10. 2025Any Information Is Just Worth One Single Screenshot: Unifying Search With Visualized Information Retrievalcited by 8
  11. 2025Benchmarking Multimodal RAG through a Chart-based Document Question-Answering Generation Frameworkcited by 8
  12. 2025FreeRet: MLLMs as Training-Free Retrieverscited by 8
  13. 2025ReMatch: Boosting Representation through Matching for Multimodal Retrievalcited by 8
  14. 2026Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrievalcited by 8
  15. 2026WeMM-Embedding: WeChat Multi-Modal Embedding Technical Reportcited by 8
  16. 2025M2IO-R1: An Efficient RL-Enhanced Reasoning Framework for Multimodal Retrieval Augmented Multimodal Generationcited by 7
  17. 2025MR2-Bench: Going Beyond Matching to Reasoning in Multimodal Retrievalcited by 7
  18. 2026TRACE: Task-Adaptive Reasoning and Representation Learning for Universal Multimodal Retrievalcited by 7
  19. 2026DeepImageSearch: Benchmarking Multimodal Agents for Context-Aware Image Retrieval in Visual Historiescited by 6
  20. 2025UniMoCo: Unified Modality Completion for Robust Multi-Modal Embeddingscited by 5
  21. 2025Highlighting What Matters: Promptable Embeddings for Attribute-Focused Image Retrievalcited by 5
  22. 2025From Generator to Embedder: Harnessing Innate Abilities of Multimodal LLMs via Building Zero-Shot Discriminative Embedding Modelcited by 5
  23. 2025OmniBridge: Unified Multimodal Understanding, Generation, and Retrieval via Latent Space Alignmentcited by 5
  24. 2025M3Retrieve: Benchmarking Multimodal Retrieval for Medicinecited by 5
  25. 2025MOON2.0: Dynamic Modality-balanced Multimodal Representation Learning for E-commerce Product Understandingcited by 5
  26. 2026ELVA: Exploring Ranking-Driven Universal Multimodal Retrievalcited by 5
  27. 2025Guiding Cross-Modal Representations with MLLM Priors via Preference Alignmentcited by 4
  28. 2026CREM: Compression-Driven Representation Enhancement for Multimodal Retrieval and Comprehensioncited by 4
  29. 2025On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Surveycited by 3
  30. 2025Explore More, Learn Better: Parallel MLLM Embeddings under Mutual Information Minimizationcited by 3
  31. 2026Magic-MM-Embedding: Towards Visual-Token-Efficient Universal Multimodal Embedding with MLLMscited by 3
  32. 2026Adaptive Global and Fine-Grained Perceptual Fusion for MLLM Embeddings Compatible with Hard Negative Amplificationcited by 3
  33. 2026ARK: A Dual-Axis Multimodal Retrieval Benchmark along Reasoning and Knowledgecited by 3
  34. 2026Efficient and High-Fidelity Omni Modality Retrievalcited by 3
  35. 2026Think When Needed: Adaptive Reasoning-Driven Multimodal Embeddings with a Dual-LoRA Architecturecited by 3
  36. 2025VIRTUE: Visual-Interactive Text-Image Universal Embeddercited by 2
  37. 2025Let Multimodal Embedders Learn When to Augment Query via Adaptive Query Augmentationcited by 2
  38. 2025Generative Giants, Retrieval Weaklings: Why do Multimodal Large Language Models Fail at Multimodal Retrieval?cited by 2

50 most cited of 84 All 84 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Indexing Multimodal Language Models for Large-scale Image Retrieval
  2. 2025IDMR: Towards Instance-Driven Precise Visual Correspondence in Multimodal Retrieval
  3. 2024Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Models
  4. 2024GME: Improving Universal Multimodal Retrieval by Multimodal LLMs
  5. 2024LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant
  6. 2025CoLLM: A Large Language Model for Composed Image Retrieval
  7. 2024INQUIRE: A Natural World Text-to-Image Retrieval Benchmark
  8. 2026Skim and Skip: Hierarchical Adaptive Inference for Efficient Multimodal Retrieval