# Keyword-Based Diverse Image Retrieval by Semantics-aware Contrastive Learning and Transformer

arXiv: https://arxiv.org/abs/2305.04072 · HTML: https://mlsanity.com/paper/2305.04072/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 May 2023.

## Built on by (most cited first, 5 of 5)

- [HLFormer: Enhancing Partially Relevant Video Retrieval with Hyperbolic Learning](https://mlsanity.com/paper/2507.17402/) (2025)
- [EvdCLIP: Improving Vision-Language Retrieval with Entity Visual Descriptions from Large Language Models](https://mlsanity.com/paper/2505.18594/) (2025)
- [COBRA: COmBinatorial Retrieval Augmentation for Few-Shot Adaptation](https://mlsanity.com/paper/2412.17684/) (2024)
- [MS-DPPs: Multi-Source Determinantal Point Processes for Contextual Diversity Refinement of Composite Attributes in Text to Image Retrieval](https://mlsanity.com/paper/2507.06654/) (2025)
- [Eliminating Hallucination in Diffusion-Augmented Interactive Text-to-Image Retrieval](https://mlsanity.com/paper/2601.20391/) (2026)

## Abstract (arXiv)

In addition to relevance, diversity is an important yet less studied performance metric of cross-modal image retrieval systems, which is critical to user experience. Existing solutions for diversity-aware image retrieval either explicitly post-process the raw retrieval results from standard retrieval systems or try to learn multi-vector representations of images to represent their diverse semantics. However, neither of them is good enough to balance relevance and diversity. On the one hand, standard retrieval systems are usually biased to common semantics and seldom exploit diversity-aware regularization in training, which makes it difficult to promote diversity by post-processing. On the other hand, multi-vector representation methods are not guaranteed to learn robust multiple projections. As a result, irrelevant images and images of rare or unique semantics may be projected inappropriately, which degrades the relevance and diversity of the results generated by some typical algorithms like top-k. To cope with these problems, this paper presents a new method called CoLT that tries to generate much more representative and robust representations for accurately classifying images. Specifically, CoLT first extracts semantics-aware image features by enhancing the preliminary representations of an existing one-to-one cross-modal system with semantics-aware contrastive learning. Then, a transformer-based token classifier is developed to subsume all the features into their corresponding categories. Finally, a post-processing algorithm is designed to retrieve images from each category to form the final retrieval result. Extensive experiments on two real-world datasets Div400 and Div150Cred show that CoLT can effectively boost diversity, and outperforms the existing methods as a whole (with a higher F1 score).