# InDiReCT: Language-Guided Zero-Shot Deep Metric Learning for Images

arXiv: https://arxiv.org/abs/2211.12760 · HTML: https://mlsanity.com/paper/2211.12760/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 36 references that are in the map.
- First posted on arXiv 23 Nov 2022.

## Builds on (most cited first, 15 of 15)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [A Metric Learning Reality Check](https://mlsanity.com/paper/2003.08505/) (2020)
- [No Fuss Distance Metric Learning using Proxies](https://mlsanity.com/paper/1703.07464/) (2017)
- [Revisiting Training Strategies and Generalization Performance in Deep Metric Learning](https://mlsanity.com/paper/2002.08473/) (2020)
- [TriMap: Large-scale Dimensionality Reduction Using Triplets](https://mlsanity.com/paper/1910.00204/) (2019)
- [From Principal Subspaces to Principal Components with Linear Autoencoders](https://mlsanity.com/paper/1804.10253/) (2018)
- [Combination of Multiple Global Descriptors for Image Retrieval](https://mlsanity.com/paper/1903.10663/) (2019)
- [Language learning using Speech to Image retrieval](https://mlsanity.com/paper/1909.03795/) (2019)
- [Characterizing Generalization under Out-Of-Distribution Shifts in Deep Metric Learning](https://mlsanity.com/paper/2107.09562/) (2021)
- [Sharing Matters for Generalization in Deep Metric Learning](https://mlsanity.com/paper/2004.05582/) (2020)
- [Evaluation Metrics for Item Recommendation under Sampling](https://mlsanity.com/paper/1912.02263/) (2019)
- [Do Different Deep Metric Learning Losses Lead to Similar Learned Features?](https://mlsanity.com/paper/2205.02698/) (2022)

## Built on by (most cited first, 5 of 5)

- [LangOcc: Self-Supervised Open Vocabulary Occupancy Estimation via Volume Rendering](https://mlsanity.com/paper/2407.17310/) (2024)
- [Robust Pedestrian Detection via Constructing Versatile Pedestrian Knowledge Bank](https://mlsanity.com/paper/2404.19299/) (2024)
- [ConceptHash: Interpretable Fine-Grained Hashing via Concept Discovery](https://mlsanity.com/paper/2406.08457/) (2024)
- [Disentangling Latent Embeddings with Sparse Linear Concept Subspaces (SLiCS)](https://mlsanity.com/paper/2508.20322/) (2025)
- [Training-free Conditional Image Embedding Framework Leveraging Large Vision Language Models](https://mlsanity.com/paper/2512.21860/) (2025)

## Abstract (arXiv)

Common Deep Metric Learning (DML) datasets specify only one notion of similarity, e.g., two images in the Cars196 dataset are deemed similar if they show the same car model. We argue that depending on the application, users of image retrieval systems have different and changing similarity notions that should be incorporated as easily as possible. Therefore, we present Language-Guided Zero-Shot Deep Metric Learning (LanZ-DML) as a new DML setting in which users control the properties that should be important for image representations without training data by only using natural language. To this end, we propose InDiReCT (Image representations using Dimensionality Reduction on CLIP embedded Texts), a model for LanZ-DML on images that exclusively uses a few text prompts for training. InDiReCT utilizes CLIP as a fixed feature extractor for images and texts and transfers the variation in text prompt embeddings to the image embedding space. Extensive experiments on five datasets and overall thirteen similarity notions show that, despite not seeing any images during training, InDiReCT performs better than strong baselines and approaches the performance of fully-supervised models. An analysis reveals that InDiReCT learns to focus on regions of the image that correlate with the desired similarity notion, which makes it a fast to train and easy to use method to create custom embedding spaces only using natural language.