# Text-Guided Image Clustering

arXiv: https://arxiv.org/abs/2402.02996 · HTML: https://mlsanity.com/paper/2402.02996/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 59 references that are in the map.
- First posted on arXiv 5 Feb 2024.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [Deep Clustering for Unsupervised Learning of Visual Features](https://mlsanity.com/paper/1807.05520/) (2018)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [Florence: A New Foundation Model for Computer Vision](https://mlsanity.com/paper/2111.11432/) (2021)
- [Unsupervised Deep Embedding for Clustering Analysis](https://mlsanity.com/paper/1511.06335/) (2015)
- [GIT: A Generative Image-to-text Transformer for Vision and Language](https://mlsanity.com/paper/2205.14100/) (2022)
- [Measuring and Improving Consistency in Pretrained Language Models](https://mlsanity.com/paper/2102.01017/) (2021)
- [Towards K-means-friendly Spaces: Simultaneous Deep Learning and Clustering](https://mlsanity.com/paper/1610.04794/) (2016)
- [Training Deep Networks for Facial Expression Recognition with Crowd-Sourced Label Distribution](https://mlsanity.com/paper/1608.01041/) (2016)
- [Explainable k-Means and k-Medians Clustering](https://mlsanity.com/paper/2002.12538/) (2020)
- [GATCluster: Self-Supervised Gaussian-Attention Network for Image Clustering](https://mlsanity.com/paper/2002.11863/) (2020)
- [Stable Cluster Discrimination for Deep Clustering](https://mlsanity.com/paper/2311.14310/) (2023)
- [Contrastive Learning of Generalized Game Representations](https://mlsanity.com/paper/2106.10060/) (2021)
- [Efficient Algorithms for Generating Provably Near-Optimal Cluster Descriptors for Explainability](https://mlsanity.com/paper/2002.02487/) (2020)
- [DeepMCAT: Large-Scale Deep Clustering for Medical Image Categorization](https://mlsanity.com/paper/2110.00109/) (2021)
- [Deep Descriptive Clustering](https://mlsanity.com/paper/2105.11549/) (2021)

## Built on by (most cited first, 5 of 5)

- [Text-Guided Alternative Image Clustering](https://mlsanity.com/paper/2406.18589/) (2024)
- [Deep Clustering via Gradual Community Detection](https://mlsanity.com/paper/2501.02036/) (2025)
- [Interpretable Text-Guided Image Clustering via Iterative Search](https://mlsanity.com/paper/2506.12514/) (2025)
- [Hierarchical Textual Knowledge for Enhanced Image Clustering](https://mlsanity.com/paper/2604.11144/) (2026)
- [GRACE: LLM-Grounded Semantic Metric Spaces for Scalable Mixed-Data Clustering](https://mlsanity.com/paper/2608.07881/) (2026)

## Abstract (arXiv)

Image clustering divides a collection of images into meaningful groups, typically interpreted post-hoc via human-given annotations. Those are usually in the form of text, begging the question of using text as an abstraction for image clustering. Current image clustering methods, however, neglect the use of generated textual descriptions. We, therefore, propose Text-Guided Image Clustering, i.e., generating text using image captioning and visual question-answering (VQA) models and subsequently clustering the generated text. Further, we introduce a novel approach to inject task- or domain knowledge for clustering by prompting VQA models. Across eight diverse image clustering datasets, our results show that the obtained text representations often outperform image features. Additionally, we propose a counting-based cluster explainability method. Our evaluations show that the derived keyword-based explanations describe clusters better than the respective cluster accuracy suggests. Overall, this research challenges traditional approaches and paves the way for a paradigm shift in image clustering, using generated text.