# Language-only Efficient Training of Zero-shot Composed Image Retrieval

arXiv: https://arxiv.org/abs/2312.01998 · HTML: https://mlsanity.com/paper/2312.01998/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 48 references that are in the map.
- First posted on arXiv 4 Dec 2023.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning](https://mlsanity.com/paper/2203.02053/) (2022)
- [A Corpus for Reasoning About Natural Language Grounded in Photographs](https://mlsanity.com/paper/1811.00491/) (2018)
- [A Metric Learning Reality Check](https://mlsanity.com/paper/2003.08505/) (2020)
- [Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Models](https://mlsanity.com/paper/2108.04024/) (2021)
- [Composing Text and Image for Image Retrieval - An Empirical Odyssey](https://mlsanity.com/paper/1812.07119/) (2018)
- [Zero-Shot Composed Image Retrieval with Textual Inversion](https://mlsanity.com/paper/2303.15247/) (2023)
- [Pic2Word: Mapping Pictures to Words for Zero-shot Composed Image Retrieval](https://mlsanity.com/paper/2302.03084/) (2023)
- [Learning to Predict Visual Attributes in the Wild](https://mlsanity.com/paper/2106.09707/) (2021)
- [Text-Only Training for Image Captioning using Noise-Injected CLIP](https://mlsanity.com/paper/2211.00575/) (2022)
- [CompoDiff: Versatile Composed Image Retrieval With Latent Diffusion](https://mlsanity.com/paper/2303.11916/) (2023)
- [DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Training](https://mlsanity.com/paper/2303.03032/) (2023)
- [ARTEMIS: Attention-based Retrieval with Text-Explicit Matching and Implicit Similarity](https://mlsanity.com/paper/2203.08101/) (2022)
- [Modality-Agnostic Attention Fusion for visual search with text feedback](https://mlsanity.com/paper/2007.00145/) (2020)
- [GeneCIS: A Benchmark for General Conditional Image Similarity](https://mlsanity.com/paper/2306.07969/) (2023)
- [Composed Image Retrieval with Text Feedback via Multi-grained Uncertainty Regularization](https://mlsanity.com/paper/2211.07394/) (2022)
- [Improved Probabilistic Image-Text Representations](https://mlsanity.com/paper/2305.18171/) (2023)
- [ECCV Caption: Correcting False Negatives by Collecting Machine-and-Human-verified Image-Caption Associations for MS-COCO](https://mlsanity.com/paper/2204.03359/) (2022)
- [RTIC: Residual Learning for Text and Image Composition using Graph Convolutional Network](https://mlsanity.com/paper/2104.03015/) (2021)

## Built on by (most cited first, 27 of 27)

- [Spherical Linear Interpolation and Text-Anchoring for Zero-shot Composed Image Retrieval](https://mlsanity.com/paper/2405.00571/) (2024)
- [Reason-before-Retrieve: One-Stage Reflective Chain-of-Thoughts for Training-Free Zero-Shot Composed Image Retrieval](https://mlsanity.com/paper/2412.11077/) (2024)
- [iSEARLE: Improving Textual Inversion for Zero-Shot Composed Image Retrieval](https://mlsanity.com/paper/2405.02951/) (2024)
- [EgoCVR: An Egocentric Benchmark for Fine-Grained Composed Video Retrieval](https://mlsanity.com/paper/2407.16658/) (2024)
- [Missing Target-Relevant Information Prediction with World Model for Accurate Zero-Shot Composed Image Retrieval](https://mlsanity.com/paper/2503.17109/) (2025)
- [Multimodal Reasoning Agent for Zero-Shot Composed Image Retrieval](https://mlsanity.com/paper/2505.19952/) (2025)
- [Towards Cross-modal Backward-compatible Representation Learning for Vision-Language Models](https://mlsanity.com/paper/2405.14715/) (2024)
- [A Survey of Multimodal Composite Editing and Retrieval](https://mlsanity.com/paper/2409.05405/) (2024)
- [MLLM-Guided VLM Fine-Tuning with Joint Inference for Zero-Shot Composed Image Retrieval](https://mlsanity.com/paper/2505.19707/) (2025)
- [Composed Image Retrieval for Training-Free Domain Conversion](https://mlsanity.com/paper/2412.03297/) (2024)
- [Multiplicity is an Inevitable and Inherent Challenge in Multimodal Learning](https://mlsanity.com/paper/2505.19614/) (2025)
- [Denoise-I2W: Mapping Images to Denoising Words for Accurate Zero-Shot Composed Image Retrieval](https://mlsanity.com/paper/2410.17393/) (2024)
- [FuseLIP: Multimodal Embeddings via Early Fusion of Discrete Tokens](https://mlsanity.com/paper/2506.03096/) (2025)
- [An Efficient Post-hoc Framework for Reducing Task Discrepancy of Text Encoders for Composed Image Retrieval](https://mlsanity.com/paper/2406.09188/) (2024)
- [G-MIXER: Geodesic Mixup-based Implicit Semantic Expansion and Explicit Semantic Re-ranking for Zero-Shot Composed Image Retrieval](https://mlsanity.com/paper/2604.14710/) (2026)
- [UniCVR: From Alignment to Reranking for Unified Zero-Shot Composed Visual Retrieval](https://mlsanity.com/paper/2604.20318/) (2026)
- [STiTch: Semantic Transition and Transportation in Collaboration for Training-Free Zero-Shot Composed Image Retrieval](https://mlsanity.com/paper/2605.21261/) (2026)
- [SETR: A Two-Stage Semantic-Enhanced Framework for Zero-Shot Composed Image Retrieval](https://mlsanity.com/paper/2509.26012/) (2025)
- [SQUARE: Semantic Query-Augmented Fusion and Efficient Batch Reranking for Training-free Zero-Shot Composed Image Retrieval](https://mlsanity.com/paper/2509.26330/) (2025)
- [Text-Only Training for Image Captioning with Retrieval Augmentation and Modality Gap Correction](https://mlsanity.com/paper/2512.04309/) (2025)
- [PREGEN: Uncovering Latent Thoughts in Composed Video Retrieval](https://mlsanity.com/paper/2601.13797/) (2026)
- [Few Shots Text to Image Retrieval: New Benchmarking Dataset and Optimization Methods](https://mlsanity.com/paper/2603.25891/) (2026)
- [Never Seen Before: Benchmarking Genuine Zero-Shot Composed Image Retrieval with Consistent Video-Sourced Datasets](https://mlsanity.com/paper/2606.07032/) (2026)
- [Generative Editing in the Joint Vision-Language Space for Zero-Shot Composed Image Retrieval](https://mlsanity.com/paper/2512.01636/) (2025)
- [A Sanity Check on Composed Image Retrieval](https://mlsanity.com/paper/2604.12904/) (2026)
- [CR-Refiner: An Object-Centric Optimal Transport Reranker for Edit-Conditioned 3D Scene Retrieval](https://mlsanity.com/paper/2607.19115/) (2026)
- [Training-Free Pseudo-Fusion for Composed Image Retrieval with Diffusion Models and Multimodal Large Language Models](https://mlsanity.com/paper/2608.23102/) (2026)

## Abstract (arXiv)

Composed image retrieval (CIR) task takes a composed query of image and text, aiming to search relative images for both conditions. Conventional CIR approaches need a training dataset composed of triplets of query image, query text, and target image, which is very expensive to collect. Several recent works have worked on the zero-shot (ZS) CIR paradigm to tackle the issue without using pre-collected triplets. However, the existing ZS-CIR methods show limited backbone scalability and generalizability due to the lack of diversity of the input texts during training. We propose a novel CIR framework, only using language for its training. Our LinCIR (Language-only training for CIR) can be trained only with text datasets by a novel self-supervision named self-masking projection (SMP). We project the text latent embedding to the token embedding space and construct a new text by replacing the keyword tokens of the original text. Then, we let the new and original texts have the same latent embedding vector. With this simple strategy, LinCIR is surprisingly efficient and highly effective; LinCIR with CLIP ViT-G backbone is trained in 48 minutes and shows the best ZS-CIR performances on four different CIR benchmarks, CIRCO, GeneCIS, FashionIQ, and CIRR, even outperforming supervised method on FashionIQ. Code is available at https://github.com/navervision/lincir