mlsanityData to 7 Oct 2026

Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Models

arXiv 2108.04024 ↗First posted 9 Aug 2021cs.CV cs.CL cs.IRcited by 244

Not on the map. Why?

Cited by 43 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We extend the task of composed image retrieval, where an input query consists of an image and short textual description of how to modify the image. Existing methods have only been applied to non-complex images within narrow domains, such as fashion products, thereby limiting the scope of study on in-depth visual reasoning in rich image and language contexts. To address this issue, we collect the Compose Image Retrieval on Real-life images (CIRR) dataset, which consists of over 36,000 pairs of crowd-sourced, open-domain images with human-generated modifying text. To extend current methods to the open-domain, we propose CIRPLANT, a transformer based model that leverages rich pre-trained vision-and-language (V&L) knowledge for modifying visual features conditioned on natural language. Retrieval is then done by nearest neighbor lookup on the modified features. We demonstrate that with a relatively simple architecture, CIRPLANT outperforms existing methods on open-domain images, while matching state-of-the-art accuracy on the existing narrow datasets, such as fashion. Together with the release of CIRR, we believe this work will inspire further research on composed image retrieval.

Key facts. Cited by 244 papers in the map (data through 7 Oct 2026). Builds on 29 of its 44 references that are in the map. First posted on arXiv 9 Aug 2021.

Builds on

29 of 44 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  5. 2015Fast R-CNNcited by 2,930
  6. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  7. 2015VQA: Visual Question Answeringcited by 2,489
  8. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  9. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  10. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  11. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  12. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
17 more
  1. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  2. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  3. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  4. 2017A simple neural network module for relational reasoningcited by 345
  5. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
  6. 2018Deep Face Recognition: A Surveycited by 275
  7. 2018Composing Text and Image for Image Retrieval - An Empirical Odysseycited by 234
  8. 2017Automatic Spatially-aware Fashion Concept Discoverycited by 132
  9. 2018Multilevel Language and Vision Integration for Text-to-Clip Retrievalcited by 128
  10. 2018Learning to Describe Differences Between Pairs of Similar Imagescited by 114
  11. 2018Dialog-based Interactive Image Retrievalcited by 84
  12. 2017Tips and Tricks for Visual Question Answering: Learnings from the 2017 Challengecited by 76
  13. 2017Convolutional Image Captioningcited by 52
  14. 2020Modality-Agnostic Attention Fusion for visual search with text feedbackcited by 51
  15. 2016Multimodal Residual Learning for Visual QAcited by 42
  16. 2019Fashion IQ: A New Dataset Towards Retrieving Images by Natural Language Feedbackcited by 42
  17. 2019Neural Naturalist: Generating Fine-Grained Image Comparisonscited by 36

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 9; 2023: 39; 2024: 50; 2025: 70; 2026: 74.

244 papers.

Sort:

By area:

  1. 2022Multimodal Learning with Transformers: A Surveycited by 399
  2. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Taskscited by 222
  3. 2022ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentationcited by 178
  4. 2023Zero-Shot Composed Image Retrieval with Textual Inversioncited by 171
  5. 2024GME: Improving Universal Multimodal Retrieval by Multimodal LLMscited by 166
  6. 2023UniIR: Training and Benchmarking Universal Multimodal Information Retrieverscited by 165
  7. 2024MM-Embed: Universal Multimodal Retrieval with Multimodal LLMscited by 159
  8. 2023Pic2Word: Mapping Pictures to Words for Zero-shot Composed Image Retrievalcited by 155
  9. 2024E5-V: Universal Embeddings with Multimodal Large Language Modelscited by 123
  10. 2024MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructionscited by 119
  11. 2024LamRA: Large Multimodal Model as Your Advanced Retrieval Assistantcited by 106
  12. 2024Jailbreak Vision Language Models via Bi-Modal Adversarial Promptcited by 103
38 more
  1. 2023Vision-by-Language for Training-Free Compositional Image Retrievalcited by 102
  2. 2023CompoDiff: Versatile Composed Image Retrieval With Latent Diffusioncited by 84
  3. 2022ARTEMIS: Attention-based Retrieval with Text-Explicit Matching and Implicit Similaritycited by 82
  4. 2024VISTA: Visualized Text Embedding For Universal Multi-Modal Retrievalcited by 67
  5. 2023Sentence-level Prompts Benefit Composed Image Retrievalcited by 59
  6. 2023Composed Image Retrieval using Contrastive Learning and Task-oriented CLIP-based Featurescited by 52
  7. 2023CoVR-2: Automatic Data Construction for Composed Video Retrievalcited by 52
  8. 2023Context-I2W: Mapping Images to Context-dependent Words for Accurate Zero-Shot Composed Image Retrievalcited by 52
  9. 2023GeneCIS: A Benchmark for General Conditional Image Similaritycited by 46
  10. 2022Composed Image Retrieval with Text Feedback via Multi-grained Uncertainty Regularizationcited by 45
  11. 2023Data Roaming and Quality Assessment for Composed Image Retrievalcited by 45
  12. 2024Spherical Linear Interpolation and Text-Anchoring for Zero-shot Composed Image Retrievalcited by 44
  13. 2023Bi-directional Training for Composed Image Retrieval via Text Prompt Learningcited by 43
  14. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  15. 2025FineCIR: Explicit Parsing of Fine-Grained Modification Semantics for Composed Image Retrievalcited by 41
  16. 2024MambaMixer: Efficient Selective State Space Models with Dual Token and Channel Selectioncited by 40
  17. 2025OFFSET: Segmentation-based Focus Shift Revision for Composed Image Retrievalcited by 40
  18. 2026ReTrack: Evidence-Driven Dual-Stream Directional Anchor Calibration Network for Composed Video Retrievalcited by 39
  19. 2021Edge-Cloud Polarization and Collaboration: A Comprehensive Survey for AIcited by 38
  20. 2023PoseFix: Correcting 3D Human Poses with Natural Languagecited by 38
  21. 2023FAME-ViL: Multi-Tasking Vision-Language Model for Heterogeneous Fashion Taskscited by 37
  22. 2025UME-R1: Exploring Reasoning-Driven Generative Multimodal Embeddingscited by 37
  23. 2024Knowledge-Enhanced Dual-stream Zero-shot Composed Image Retrievalcited by 36
  24. 2023Candidate Set Re-ranking for Composed Image Retrieval with Dual Multi-modal Encodercited by 35
  25. 2022FashionViL: Fashion-Focused Vision-and-Language Representation Learningcited by 32
  26. 2025HUD: Hierarchical Uncertainty-Aware Disambiguation Network for Composed Video Retrievalcited by 32
  27. 2023Zero-shot Composed Text-Image Retrievalcited by 31
  28. 2024Universal Adversarial Perturbations for Vision-Language Pre-trained Modelscited by 31
  29. 2024Reason-before-Retrieve: One-Stage Reflective Chain-of-Thoughts for Training-Free Zero-Shot Composed Image Retrievalcited by 31
  30. 2023Language-only Efficient Training of Zero-shot Composed Image Retrievalcited by 27
  31. 2023Image Clustering Conditioned on Text Criteriacited by 26
  32. 2024Composed Video Retrieval via Enriched Context and Discriminative Embeddingscited by 26
  33. 2025Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrievalcited by 26
  34. 2026INTENT: Invariance and Discrimination-aware Noise Mitigation for Robust Composed Image Retrievalcited by 26
  35. 2023OpenVIS: Open-vocabulary Video Instance Segmentationcited by 25
  36. 2023Chatting Makes Perfect: Chat-based Image Retrievalcited by 25
  37. 2024Interactive Text-to-Image Retrieval with Large Language Models: A Plug-and-Play Approachcited by 25
  38. 2024iSEARLE: Improving Textual Inversion for Zero-Shot Composed Image Retrievalcited by 24

50 most cited of 244 All 244 (JSON)