# From Mapping to Composing: A Two-Stage Framework for Zero-shot Composed Image Retrieval

arXiv: https://arxiv.org/abs/2504.17990 · HTML: https://mlsanity.com/paper/2504.17990/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 56 references that are in the map.
- First posted on arXiv 25 Apr 2025.

## Builds on (most cited first, 30 of 35)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Corpus for Reasoning About Natural Language Grounded in Photographs](https://mlsanity.com/paper/1811.00491/) (2018)
- [Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Models](https://mlsanity.com/paper/2108.04024/) (2021)
- [Composing Text and Image for Image Retrieval - An Empirical Odyssey](https://mlsanity.com/paper/1812.07119/) (2018)
- [Zero-Shot Composed Image Retrieval with Textual Inversion](https://mlsanity.com/paper/2303.15247/) (2023)
- [Pic2Word: Mapping Pictures to Words for Zero-shot Composed Image Retrieval](https://mlsanity.com/paper/2302.03084/) (2023)
- [Vision-by-Language for Training-Free Compositional Image Retrieval](https://mlsanity.com/paper/2310.09291/) (2023)
- [CompoDiff: Versatile Composed Image Retrieval With Latent Diffusion](https://mlsanity.com/paper/2303.11916/) (2023)
- [Context-I2W: Mapping Images to Context-dependent Words for Accurate Zero-Shot Composed Image Retrieval](https://mlsanity.com/paper/2309.16137/) (2023)
- [Spherical Linear Interpolation and Text-Anchoring for Zero-shot Composed Image Retrieval](https://mlsanity.com/paper/2405.00571/) (2024)
- [Target-Guided Composed Image Retrieval](https://mlsanity.com/paper/2309.01366/) (2023)
- [Knowledge-Enhanced Dual-stream Zero-shot Composed Image Retrieval](https://mlsanity.com/paper/2403.16005/) (2024)
- [Reading-strategy Inspired Visual Representation Learning for Text-to-Video Retrieval](https://mlsanity.com/paper/2201.09168/) (2022)
- [Zero-shot Composed Text-Image Retrieval](https://mlsanity.com/paper/2306.07272/) (2023)
- [Reason-before-Retrieve: One-Stage Reflective Chain-of-Thoughts for Training-Free Zero-Shot Composed Image Retrieval](https://mlsanity.com/paper/2412.11077/) (2024)
- [A Large Cross-Modal Video Retrieval Dataset with Reading Comprehension](https://mlsanity.com/paper/2305.03347/) (2023)
- [Progressive Learning for Image Retrieval with Hybrid-Modality Queries](https://mlsanity.com/paper/2204.11212/) (2022)
- [Training-free Zero-shot Composed Image Retrieval with Local Concept Reranking](https://mlsanity.com/paper/2312.08924/) (2023)
- [Cross-Lingual Cross-Modal Retrieval with Noise-Robust Learning](https://mlsanity.com/paper/2208.12526/) (2022)
- [CaLa: Complementary Association Learning for Augmenting Composed Image Retrieval](https://mlsanity.com/paper/2405.19149/) (2024)
- [Improving Composed Image Retrieval via Contrastive Learning with Scaling Positives and Negatives](https://mlsanity.com/paper/2404.11317/) (2024)
- [Missing Target-Relevant Information Prediction with World Model for Accurate Zero-Shot Composed Image Retrieval](https://mlsanity.com/paper/2503.17109/) (2025)
- [Fine-grained Textual Inversion Network for Zero-Shot Composed Image Retrieval](https://mlsanity.com/paper/2503.19296/) (2025)
- [CoTMR: Chain-of-Thought Multi-Scale Reasoning for Training-Free Zero-Shot Composed Image Retrieval](https://mlsanity.com/paper/2502.20826/) (2025)
- [Dynamic Weighted Combiner for Mixed-Modal Image Retrieval](https://mlsanity.com/paper/2312.06179/) (2023)
- [Image2Sentence based Asymmetrical Zero-shot Composed Image Retrieval](https://mlsanity.com/paper/2403.01431/) (2024)
- [Multimodal LLM Enhanced Cross-lingual Cross-modal Retrieval](https://mlsanity.com/paper/2409.19961/) (2024)
- [Efficient Token-Guided Image-Text Retrieval with Consistent Multimodal Contrastive Training](https://mlsanity.com/paper/2306.08789/) (2023)
- [CL2CM: Improving Cross-Lingual Cross-Modal Retrieval via Cross-Lingual Knowledge Transfer](https://mlsanity.com/paper/2312.08984/) (2023)
- [Dual-view Curricular Optimal Transport for Cross-lingual Cross-modal Retrieval](https://mlsanity.com/paper/2309.05451/) (2023)

## Built on by (most cited first, 2 of 2)

- [HumanSense: From Multimodal Perception to Empathetic Context-Aware Responses through Reasoning MLLMs](https://mlsanity.com/paper/2508.10576/) (2025)
- [Thinking Before Retrieving: Robust Zero-Shot Composed Image Retrieval via Strategic Planning and Self-Criticism](https://mlsanity.com/paper/2606.31222/) (2026)

## Abstract (arXiv)

Composed Image Retrieval (CIR) is a challenging multimodal task that retrieves a target image based on a reference image and accompanying modification text. Due to the high cost of annotating CIR triplet datasets, zero-shot (ZS) CIR has gained traction as a promising alternative. Existing studies mainly focus on projection-based methods, which map an image to a single pseudo-word token. However, these methods face three critical challenges: (1) insufficient pseudo-word token representation capacity, (2) discrepancies between training and inference phases, and (3) reliance on large-scale synthetic data. To address these issues, we propose a two-stage framework where the training is accomplished from mapping to composing. In the first stage, we enhance image-to-pseudo-word token learning by introducing a visual semantic injection module and a soft text alignment objective, enabling the token to capture richer and fine-grained image information. In the second stage, we optimize the text encoder using a small amount of synthetic triplet data, enabling it to effectively extract compositional semantics by combining pseudo-word tokens with modification text for accurate target image retrieval. The strong visual-to-pseudo mapping established in the first stage provides a solid foundation for the second stage, making our approach compatible with both high- and low-quality synthetic data, and capable of achieving significant performance gains with only a small amount of synthetic data. Extensive experiments were conducted on three public datasets, achieving superior performance compared to existing approaches.