# Image Clustering Conditioned on Text Criteria

arXiv: https://arxiv.org/abs/2310.18297 · HTML: https://mlsanity.com/paper/2310.18297/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 53 of its 84 references that are in the map.
- First posted on arXiv 27 Oct 2023.

## Builds on (most cited first, 30 of 53)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](https://mlsanity.com/paper/2112.06905/) (2021)
- [Self-Supervised Learning of Pretext-Invariant Representations](https://mlsanity.com/paper/1912.01991/) (2019)
- [ClipCap: CLIP Prefix for Image Captioning](https://mlsanity.com/paper/2111.09734/) (2021)
- [Otter: A Multi-Modal Model with In-Context Instruction Tuning](https://mlsanity.com/paper/2305.03726/) (2023)
- [Invariant Information Clustering for Unsupervised Image Classification and Segmentation](https://mlsanity.com/paper/1807.06653/) (2018)
- [SCAN: Learning to Classify Images without Labels](https://mlsanity.com/paper/2005.12320/) (2020)
- [Visual Classification via Description from Large Language Models](https://mlsanity.com/paper/2210.07183/) (2022)
- [D2-Net: A Trainable CNN for Joint Detection and Description of Local Features](https://mlsanity.com/paper/1905.03561/) (2019)
- [LIFT: Learned Invariant Feature Transform](https://mlsanity.com/paper/1603.09114/) (2016)
- [GLIPv2: Unifying Localization and Vision-Language Understanding](https://mlsanity.com/paper/2206.05836/) (2022)
- [MultiModal-GPT: A Vision and Language Model for Dialogue with Humans](https://mlsanity.com/paper/2305.04790/) (2023)
- [Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Models](https://mlsanity.com/paper/2108.04024/) (2021)
- [Large-Scale Image Retrieval with Attentive Deep Local Features](https://mlsanity.com/paper/1612.06321/) (2016)

## Built on by (most cited first, 26 of 26)

- [LLM as Dataset Analyst: Subpopulation Structure Discovery with Large Language Model](https://mlsanity.com/paper/2405.02363/) (2024)
- [Text Clustering as Classification with LLMs](https://mlsanity.com/paper/2410.00927/) (2024)
- [Organizing Unstructured Image Collections using Natural Language](https://mlsanity.com/paper/2410.05217/) (2024)
- [Generative Timelines for Instructed Visual Assembly](https://mlsanity.com/paper/2411.12293/) (2024)
- [Conditional Representation Learning for Customized Tasks](https://mlsanity.com/paper/2510.04564/) (2025)
- [ConceptScope: Characterizing Dataset Bias via Disentangled Visual Concepts](https://mlsanity.com/paper/2510.26186/) (2025)
- [VLMine: Long-Tail Data Mining with Vision Language Models](https://mlsanity.com/paper/2409.15486/) (2024)
- [Deep Spectral Clustering via Joint Spectral Embedding and Kmeans](https://mlsanity.com/paper/2412.11080/) (2024)
- [HERCULES: Hierarchical Embedding-based Recursive Clustering Using LLMs for Efficient Summarization](https://mlsanity.com/paper/2506.19992/) (2025)
- [Open Ad-hoc Categorization with Contextualized Feature Learning](https://mlsanity.com/paper/2512.16202/) (2025)
- [The Many Senses of Visual Similarity: A Text-Prompted Image Perceptual Metric](https://mlsanity.com/paper/2607.18237/) (2026)
- [Bootstrap Deep Spectral Clustering with Optimal Transport](https://mlsanity.com/paper/2508.04200/) (2025)
- [Text-Guided Alternative Image Clustering](https://mlsanity.com/paper/2406.18589/) (2024)
- [CLAY: Conditional Visual Similarity Modulation in Vision-Language Embedding Space](https://mlsanity.com/paper/2604.11539/) (2026)
- [Dual-Level Cross-Modal Contrastive Clustering](https://mlsanity.com/paper/2409.04561/) (2024)
- [DSAI: Unbiased and Interpretable Latent Feature Extraction for Data-Centric AI](https://mlsanity.com/paper/2412.06303/) (2024)
- [Deep Clustering via Gradual Community Detection](https://mlsanity.com/paper/2501.02036/) (2025)
- [Learning a Canonical Basis of Human Preferences from Binary Ratings](https://mlsanity.com/paper/2503.24150/) (2025)
- [Interpretable Text-Guided Image Clustering via Iterative Search](https://mlsanity.com/paper/2506.12514/) (2025)
- [In-Context Clustering with Large Language Models](https://mlsanity.com/paper/2510.08466/) (2025)
- [GH-ESD: Grounded Hypothesis-Driven Error Slice Discovery for Instance-Level Vision Tasks](https://mlsanity.com/paper/2512.24592/) (2025)
- [Semantic Purification for Conditional Representation Learning](https://mlsanity.com/paper/2602.05464/) (2026)
- [Language-Assisted Image Clustering Guided by Discriminative Relational Signals and Adaptive Semantic Centers](https://mlsanity.com/paper/2603.24275/) (2026)
- [Hierarchical Textual Knowledge for Enhanced Image Clustering](https://mlsanity.com/paper/2604.11144/) (2026)
- [Universal Guideline-Driven Image Clustering via a Hybrid LLM Agent](https://mlsanity.com/paper/2606.24094/) (2026)
- [Controlling Embedding Spaces with Text-Conditioned Transformations](https://mlsanity.com/paper/2607.22919/) (2026)

## Abstract (arXiv)

Classical clustering methods do not provide users with direct control of the clustering results, and the clustering results may not be consistent with the relevant criterion that a user has in mind. In this work, we present a new methodology for performing image clustering based on user-specified text criteria by leveraging modern vision-language models and large language models. We call our method Image Clustering Conditioned on Text Criteria (IC|TC), and it represents a different paradigm of image clustering. IC|TC requires a minimal and practical degree of human intervention and grants the user significant control over the clustering results in return. Our experiments show that IC|TC can effectively cluster images with various criteria, such as human action, physical location, or the person's mood, while significantly outperforming baselines.