# Zero-Shot Composed Image Retrieval with Textual Inversion

arXiv: https://arxiv.org/abs/2303.15247 · HTML: https://mlsanity.com/paper/2303.15247/

## Key facts

- Cited by 176 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 44 references that are in the map.
- First posted on arXiv 27 Mar 2023.

## Builds on (most cited first, 30 of 31)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [Multi-Concept Customization of Text-to-Image Diffusion](https://mlsanity.com/paper/2212.04488/) (2022)
- [Designing an Encoder for StyleGAN Image Manipulation](https://mlsanity.com/paper/2102.02766/) (2021)
- [In-Domain GAN Inversion for Real Image Editing](https://mlsanity.com/paper/2004.00049/) (2020)
- [Meshed-Memory Transformer for Image Captioning](https://mlsanity.com/paper/1912.08226/) (2019)
- [Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Models](https://mlsanity.com/paper/2108.04024/) (2021)
- [Composing Text and Image for Image Retrieval - An Empirical Odyssey](https://mlsanity.com/paper/1812.07119/) (2018)
- [Scaling Up Vision-Language Pre-training for Image Captioning](https://mlsanity.com/paper/2111.12233/) (2021)
- [Pic2Word: Mapping Pictures to Words for Zero-shot Composed Image Retrieval](https://mlsanity.com/paper/2302.03084/) (2023)
- [Automatic Spatially-aware Fashion Concept Discovery](https://mlsanity.com/paper/1708.01311/) (2017)
- ["This is my unicorn, Fluffy": Personalizing frozen vision-language representations](https://mlsanity.com/paper/2204.01694/) (2022)
- [Dialog-based Interactive Image Retrieval](https://mlsanity.com/paper/1805.00145/) (2018)
- [ARTEMIS: Attention-based Retrieval with Text-Explicit Matching and Implicit Similarity](https://mlsanity.com/paper/2203.08101/) (2022)
- [Neural Naturalist: Generating Fine-Grained Image Comparisons](https://mlsanity.com/paper/1909.04101/) (2019)
- [Multiresolution Textual Inversion](https://mlsanity.com/paper/2211.17115/) (2022)

## Built on by (most cited first, 30 of 176)

- [MM-Embed: Universal Multimodal Retrieval with Multimodal LLMs](https://mlsanity.com/paper/2411.02571/) (2024)
- [LaDI-VTON: Latent Diffusion Textual-Inversion Enhanced Virtual Try-On](https://mlsanity.com/paper/2305.13501/) (2023)
- [MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructions](https://mlsanity.com/paper/2403.19651/) (2024)
- [LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant](https://mlsanity.com/paper/2412.01720/) (2024)
- [Vision-by-Language for Training-Free Compositional Image Retrieval](https://mlsanity.com/paper/2310.09291/) (2023)
- [Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMs](https://mlsanity.com/paper/2504.17432/) (2025)
- [Sentence-level Prompts Benefit Composed Image Retrieval](https://mlsanity.com/paper/2310.05473/) (2023)
- [MyVLM: Personalizing VLMs for User-Specific Queries](https://mlsanity.com/paper/2403.14599/) (2024)
- [A Review of Modern Recommender Systems Using Generative Models (Gen-RecSys)](https://mlsanity.com/paper/2404.00579/) (2024)
- [CoVR-2: Automatic Data Construction for Composed Video Retrieval](https://mlsanity.com/paper/2308.14746/) (2023)
- [Context-I2W: Mapping Images to Context-dependent Words for Accurate Zero-Shot Composed Image Retrieval](https://mlsanity.com/paper/2309.16137/) (2023)
- [Spherical Linear Interpolation and Text-Anchoring for Zero-shot Composed Image Retrieval](https://mlsanity.com/paper/2405.00571/) (2024)
- [Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion](https://mlsanity.com/paper/2502.04263/) (2025)
- [Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation](https://mlsanity.com/paper/2502.08826/) (2025)
- [Knowledge-Enhanced Dual-stream Zero-shot Composed Image Retrieval](https://mlsanity.com/paper/2403.16005/) (2024)
- [Zero-shot Composed Text-Image Retrieval](https://mlsanity.com/paper/2306.07272/) (2023)
- [Reason-before-Retrieve: One-Stage Reflective Chain-of-Thoughts for Training-Free Zero-Shot Composed Image Retrieval](https://mlsanity.com/paper/2412.11077/) (2024)
- [IDEA: Inverted Text with Cooperative Deformable Aggregation for Multi-modal Object Re-Identification](https://mlsanity.com/paper/2503.10324/) (2025)
- [Language-only Efficient Training of Zero-shot Composed Image Retrieval](https://mlsanity.com/paper/2312.01998/) (2023)
- [iSEARLE: Improving Textual Inversion for Zero-Shot Composed Image Retrieval](https://mlsanity.com/paper/2405.02951/) (2024)
- [Training-free Zero-shot Composed Image Retrieval with Local Concept Reranking](https://mlsanity.com/paper/2312.08924/) (2023)
- [EgoCVR: An Egocentric Benchmark for Fine-Grained Composed Video Retrieval](https://mlsanity.com/paper/2407.16658/) (2024)
- [CoLLM: A Large Language Model for Composed Image Retrieval](https://mlsanity.com/paper/2503.19910/) (2025)
- [Mapping Memes to Words for Multimodal Hateful Meme Classification](https://mlsanity.com/paper/2310.08368/) (2023)
- [FreestyleRet: Retrieving Images from Style-Diversified Queries](https://mlsanity.com/paper/2312.02428/) (2023)
- [Improving Composed Image Retrieval via Contrastive Learning with Scaling Positives and Negatives](https://mlsanity.com/paper/2404.11317/) (2024)
- [Improving Zero-shot Generalization of Learned Prompts via Unsupervised Knowledge Distillation](https://mlsanity.com/paper/2407.03056/) (2024)
- [Imagine and Seek: Improving Composed Image Retrieval with an Imagined Proxy](https://mlsanity.com/paper/2411.16752/) (2024)
- [Missing Target-Relevant Information Prediction with World Model for Accurate Zero-Shot Composed Image Retrieval](https://mlsanity.com/paper/2503.17109/) (2025)
- [Fine-grained Textual Inversion Network for Zero-Shot Composed Image Retrieval](https://mlsanity.com/paper/2503.19296/) (2025)

## Abstract (arXiv)

Composed Image Retrieval (CIR) aims to retrieve a target image based on a query composed of a reference image and a relative caption that describes the difference between the two images. The high effort and cost required for labeling datasets for CIR hamper the widespread usage of existing methods, as they rely on supervised learning. In this work, we propose a new task, Zero-Shot CIR (ZS-CIR), that aims to address CIR without requiring a labeled training dataset. Our approach, named zero-Shot composEd imAge Retrieval with textuaL invErsion (SEARLE), maps the visual features of the reference image into a pseudo-word token in CLIP token embedding space and integrates it with the relative caption. To support research on ZS-CIR, we introduce an open-domain benchmarking dataset named Composed Image Retrieval on Common Objects in context (CIRCO), which is the first dataset for CIR containing multiple ground truths for each query. The experiments show that SEARLE exhibits better performance than the baselines on the two main datasets for CIR tasks, FashionIQ and CIRR, and on the proposed CIRCO. The dataset, the code and the model are publicly available at https://github.com/miccunifi/SEARLE.