# Multi-modal Reference Learning for Fine-grained Text-to-Image Retrieval

arXiv: https://arxiv.org/abs/2504.07718 · HTML: https://mlsanity.com/paper/2504.07718/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 78 references that are in the map.
- First posted on arXiv 10 Apr 2025.

## Builds on (most cited first, 30 of 48)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [FILIP: Fine-grained Interactive Language-Image Pre-Training](https://mlsanity.com/paper/2111.07783/) (2021)
- [ClipCap: CLIP Prefix for Image Captioning](https://mlsanity.com/paper/2111.09734/) (2021)
- [Person Transfer GAN to Bridge Domain Gap for Person Re-Identification](https://mlsanity.com/paper/1711.08565/) (2017)
- [Bag of Tricks and A Strong Baseline for Deep Person Re-identification](https://mlsanity.com/paper/1903.07071/) (2019)
- [Stacked Cross Attention for Image-Text Matching](https://mlsanity.com/paper/1803.08024/) (2018)
- [Two at Once: Enhancing Learning and Generalization Capacities via IBN-Net](https://mlsanity.com/paper/1807.09441/) (2018)
- [No Fuss Distance Metric Learning using Proxies](https://mlsanity.com/paper/1703.07464/) (2017)
- [Proxy Anchor Loss for Deep Metric Learning](https://mlsanity.com/paper/2003.13911/) (2020)
- [Learning Deep Representations of Fine-grained Visual Descriptions](https://mlsanity.com/paper/1605.05395/) (2016)
- [Person Search with Natural Language Description](https://mlsanity.com/paper/1702.05729/) (2017)
- [Dual-Path Convolutional Image-Text Embeddings with Instance Loss](https://mlsanity.com/paper/1711.05535/) (2017)
- [Style Normalization and Restitution for Generalizable Person Re-identification](https://mlsanity.com/paper/2005.11037/) (2020)
- [Semantically Self-Aligned Network for Text-to-Image Part-aware Person Re-identification](https://mlsanity.com/paper/2107.12666/) (2021)
- [ProxyNCA++: Revisiting and Revitalizing Proxy Neighborhood Component Analysis](https://mlsanity.com/paper/2004.01113/) (2020)
- [DSSL: Deep Surroundings-person Separation Learning for Text-based Person Retrieval](https://mlsanity.com/paper/2109.05534/) (2021)
- [CLIP-Driven Fine-grained Text-Image Person Re-identification](https://mlsanity.com/paper/2210.10276/) (2022)
- [ViTAA: Visual-Textual Attributes Alignment in Person Search by Natural Language](https://mlsanity.com/paper/2005.07327/) (2020)
- [CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignment](https://mlsanity.com/paper/2209.06430/) (2022)
- [Adversarial Representation Learning for Text-to-Image Matching](https://mlsanity.com/paper/1908.10534/) (2019)
- [See Finer, See More: Implicit Modality Alignment for Text-based Person Retrieval](https://mlsanity.com/paper/2208.08608/) (2022)
- [Improving Description-based Person Re-identification by Multi-granularity Image-text Alignments](https://mlsanity.com/paper/1906.09610/) (2019)
- [Contextual Non-Local Alignment over Full-Scale Representation for Text-Based Person Search](https://mlsanity.com/paper/2101.03036/) (2021)

## Built on by (most cited first, 6 of 6)

- [Hierarchical Prompt Learning for Image- and Text-Based Person Re-Identification](https://mlsanity.com/paper/2511.13575/) (2025)
- [LLM-Driven Completeness and Consistency Evaluation for Cultural Heritage Data Augmentation in Cross-Modal Retrieval](https://mlsanity.com/paper/2511.06268/) (2025)
- [DIR-TIR: Dialog-Iterative Refinement for Text-to-Image Retrieval](https://mlsanity.com/paper/2511.14449/) (2025)
- [Fast-then-Fine: A Two-Stage Framework with Multi-Granular Representation for Cross-Modal Retrieval in Remote Sensing](https://mlsanity.com/paper/2604.20429/) (2026)
- [DREAM: Extending Vision-Language Models with Dual-Objective Encoding for Cross-Modal Retrieval](https://mlsanity.com/paper/2606.19062/) (2026)
- [Cross-Resolution Semantic Transfer for Robust Text-to-Image Retrieval in Low-Resolution Surveillance](https://mlsanity.com/paper/2606.30458/) (2026)

## Abstract (arXiv)

Fine-grained text-to-image retrieval aims to retrieve a fine-grained target image with a given text query. Existing methods typically assume that each training image is accurately depicted by its textual descriptions. However, textual descriptions can be ambiguous and fail to depict discriminative visual details in images, leading to inaccurate representation learning. To alleviate the effects of text ambiguity, we propose a Multi-Modal Reference learning framework to learn robust representations. We first propose a multi-modal reference construction module to aggregate all visual and textual details of the same object into a comprehensive multi-modal reference. The multi-modal reference hence facilitates the subsequent representation learning and retrieval similarity computation. Specifically, a reference-guided representation learning module is proposed to use multi-modal references to learn more accurate visual and textual representations. Additionally, we introduce a reference-based refinement method that employs the object references to compute a reference-based similarity that refines the initial retrieval results. Extensive experiments are conducted on five fine-grained text-to-image retrieval datasets for different text-to-image retrieval tasks. The proposed method has achieved superior performance over state-of-the-art methods. For instance, on the text-to-person image retrieval dataset RSTPReid, our method achieves the Rank1 accuracy of 56.2%, surpassing the recent CFine by 5.6%.