mlsanityData to 14 Oct 2026

MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructions

arXiv 2403.19651 ↗First posted 28 Mar 2024cs.CV cs.AI cs.CLcited by 119

Not on the map. Why?

Cited by 31 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Image retrieval, i.e., finding desired images given a reference image, inherently encompasses rich, multi-faceted search intents that are difficult to capture solely using image-based measures. Recent works leverage text instructions to allow users to more freely express their search intents. However, they primarily focus on image pairs that are visually similar and/or can be characterized by a small set of pre-defined relations. The core thesis of this paper is that text instructions can enable retrieving images with richer relations beyond visual similarity. To show this, we introduce MagicLens, a series of self-supervised image retrieval models that support open-ended instructions. MagicLens is built on a key novel insight: image pairs that naturally occur on the same web pages contain a wide range of implicit relations (e.g., inside view of), and we can bring those implicit relations explicit by synthesizing instructions via foundation models. Trained on 36.7M (query image, instruction, target image) triplets with rich semantic relations mined from the web, MagicLens achieves results comparable with or better than prior best on eight benchmarks of various image retrieval tasks, while maintaining high parameter efficiency with a significantly smaller model size. Additional human analyses on a 1.4M-image unseen corpus further demonstrate the diversity of search intents supported by MagicLens. Code and models are publicly available at https://open-vision-language.github.io/MagicLens/.

Key facts. Cited by 119 papers in the map (data through 14 Oct 2026). Builds on 46 of its 57 references that are in the map. First posted on arXiv 28 Mar 2024.

Builds on

46 of 57 references.

References · 46

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  8. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  11. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  12. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
34 more
  1. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  2. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  3. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  4. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  5. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  6. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  7. 2021Scaling Vision Transformerscited by 999
  8. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  9. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  10. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 459
  11. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 333
  12. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 255
  13. 2021Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Modelscited by 251
  14. 2018Composing Text and Image for Image Retrieval - An Empirical Odysseycited by 236
  15. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Textcited by 220
  16. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 207
  17. 2023Zero-Shot Composed Image Retrieval with Textual Inversioncited by 176
  18. 2023UniIR: Training and Benchmarking Universal Multimodal Information Retrieverscited by 166
  19. 2023Pic2Word: Mapping Pictures to Words for Zero-shot Composed Image Retrievalcited by 156
  20. 2016Deep Image Retrieval: Learning global representations for image searchcited by 147
  21. 2020Learning the Best Pooling Strategy for Visual Semantic Embeddingcited by 137
  22. 2022REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memorycited by 123
  23. 2023Open-domain Visual Entity Recognition: Towards Recognizing Millions of Wikipedia Entitiescited by 113
  24. 2022Task-aware Retrieval with Instructionscited by 107
  25. 2023Vision-by-Language for Training-Free Compositional Image Retrievalcited by 103
  26. 2022"This is my unicorn, Fluffy": Personalizing frozen vision-language representationscited by 95
  27. 2023CompoDiff: Versatile Composed Image Retrieval With Latent Diffusioncited by 85
  28. 2019Doodle to Search: Practical Zero-Shot Sketch-based Image Retrievalcited by 79
  29. 2018A Zero-Shot Framework for Sketch-based Image Retrievalcited by 71
  30. 2023Context-I2W: Mapping Images to Context-dependent Words for Accurate Zero-Shot Composed Image Retrievalcited by 52
  31. 2023GeneCIS: A Benchmark for General Conditional Image Similaritycited by 47
  32. 2019Semantic-Aware Knowledge Preservation for Zero-Shot Sketch-Based Image Retrievalcited by 40
  33. 2023Zero-Shot Everything Sketch-Based Image Retrieval, and in Explainable Stylecited by 25
  34. 2023Pretrain like Your Inference: Masked Tuning Improves Zero-Shot Composed Image Retrievalcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 14; 2025: 52; 2026: 53.

119 papers.

Sort:

By area:

  1. 2025Transfer between Modalities with MetaQueriescited by 236
  2. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Taskscited by 228
  3. 2024MM-Embed: Universal Multimodal Retrieval with Multimodal LLMscited by 161
  4. 2024LamRA: Large Multimodal Model as Your Advanced Retrieval Assistantcited by 106
  5. 2024MegaPairs: Massive Data Synthesis For Universal Multimodal Retrievalcited by 84
  6. 2025Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMscited by 73
  7. 2024TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negativescited by 58
  8. 2025LLaVE: Large Language and Vision Embedding Models with Hardness-Weighted Contrastive Learningcited by 55
  9. 2025mmE5: Improving Multimodal Multilingual Embeddings via High-quality Synthetic Datacited by 49
  10. 2024MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Modelscited by 42
  11. 2025UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding Learningcited by 42
  12. 2025MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interactioncited by 39
38 more
  1. 2025UME-R1: Exploring Reasoning-Driven Generative Multimodal Embeddingscited by 38
  2. 2024Reason-before-Retrieve: One-Stage Reflective Chain-of-Thoughts for Training-Free Zero-Shot Composed Image Retrievalcited by 32
  3. 2025CAFe: Unifying Representation and Generation with Contrastive-Autoregressive Finetuningcited by 31
  4. 2025RzenEmbed: Towards Comprehensive Multimodal Retrievalcited by 29
  5. 2025Breaking the Batch Barrier (B3) of Contrastive Learning via Smart Batch Miningcited by 28
  6. 2025Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrievalcited by 26
  7. 2026Embed-RL: Reinforcement Learning for Reasoning-Driven Multimodal Embeddingscited by 24
  8. 2025Improve Multi-Modal Embedding Learning via Explicit Hard Negative Gradient Amplifyingcited by 23
  9. 2026jina-embeddings-v5-text: Task-Targeted Embedding Distillationcited by 22
  10. 2025CoLLM: A Large Language Model for Composed Image Retrievalcited by 21
  11. 2025MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddingscited by 20
  12. 2024Imagine and Seek: Improving Composed Image Retrieval with an Imagined Proxycited by 18
  13. 2025Missing Target-Relevant Information Prediction with World Model for Accurate Zero-Shot Composed Image Retrievalcited by 18
  14. 2025A Comprehensive Survey on Composed Image Retrievalcited by 17
  15. 2026TEMA: Anchor the Image, Follow the Text for Multi-Modification Composed Image Retrievalcited by 15
  16. 2024Training-free Zero-shot Composed Image Retrieval via Weighted Modality Fusion and Similaritycited by 12
  17. 2025MRMR: A Realistic and Expert-Level Multidisciplinary Benchmark for Reasoning-Intensive Multimodal Retrievalcited by 12
  18. 2025Instance-Level Composed Image Retrievalcited by 12
  19. 2026PLUME: Latent Reasoning Based Universal Multimodal Embeddingcited by 11
  20. 2026COMBINER: Composed Image Retrieval Guided by Attribute-based Neighbor Relationscited by 11
  21. 2025U-MARVEL: Unveiling Key Factors for Universal Multimodal Retrieval via Embedding Learning with MLLMscited by 10
  22. 2025Reasoning Guided Embeddings: Leveraging MLLM Reasoning for Improved Multimodal Retrievalcited by 9
  23. 2025Any Information Is Just Worth One Single Screenshot: Unifying Search With Visualized Information Retrievalcited by 8
  24. 2025IDMR: Towards Instance-Driven Precise Visual Correspondence in Multimodal Retrievalcited by 8
  25. 2025QuRe: Query-Relevant Retrieval through Hard Negative Sampling in Composed Image Retrievalcited by 8
  26. 2025ReMatch: Boosting Representation through Matching for Multimodal Retrievalcited by 8
  27. 2025Scale Up Composed Image Retrieval Learning via Modification Text Generationcited by 7
  28. 2025ConText-CIR: Learning from Concepts in Text for Composed Image Retrievalcited by 7
  29. 2025MR2-Bench: Going Beyond Matching to Reasoning in Multimodal Retrievalcited by 7
  30. 2026TRACE: Task-Adaptive Reasoning and Representation Learning for Universal Multimodal Retrievalcited by 7
  31. 2025Joint Fusion and Encoding: Advancing Multimodal Retrieval from the Ground Upcited by 6
  32. 2026DeepImageSearch: Benchmarking Multimodal Agents for Context-Aware Image Retrieval in Visual Historiescited by 6
  33. 2026TSEmbed: Unlocking Task Scaling in Universal Multimodal Embeddingscited by 6
  34. 2026MMEmb-R1: Reasoning-Enhanced Multimodal Embedding with Pair-Aware Selection and Adaptive Controlcited by 6
  35. 2024A Survey of Multimodal Composite Editing and Retrievalcited by 5
  36. 2025Highlighting What Matters: Promptable Embeddings for Attribute-Focused Image Retrievalcited by 5
  37. 2025From Generator to Embedder: Harnessing Innate Abilities of Multimodal LLMs via Building Zero-Shot Discriminative Embedding Modelcited by 5
  38. 2025OmniBridge: Unified Multimodal Understanding, Generation, and Retrieval via Latent Space Alignmentcited by 5

50 most cited of 119 All 119 (JSON)