# Hierarchical Semantic Alignment for Image Clustering

arXiv: https://arxiv.org/abs/2512.00904 · HTML: https://mlsanity.com/paper/2512.00904/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 45 references that are in the map.
- First posted on arXiv 30 Nov 2025.

## Builds on (most cited first, 26 of 26)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [Unsupervised Deep Embedding for Clustering Analysis](https://mlsanity.com/paper/1511.06335/) (2015)
- [SCAN: Learning to Classify Images without Labels](https://mlsanity.com/paper/2005.12320/) (2020)
- [Contrastive Clustering](https://mlsanity.com/paper/2009.09687/) (2020)
- [Towards K-means-friendly Spaces: Simultaneous Deep Learning and Clustering](https://mlsanity.com/paper/1610.04794/) (2016)
- [Joint Unsupervised Learning of Deep Representations and Image Clusters](https://mlsanity.com/paper/1604.03628/) (2016)
- [Learning Discrete Representations via Information Maximizing Self-Augmented Training](https://mlsanity.com/paper/1702.08720/) (2017)
- [Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)](https://mlsanity.com/paper/2205.01397/) (2022)
- [SPICE: Semantic Pseudo-labeling for Image Clustering](https://mlsanity.com/paper/2103.09382/) (2021)
- [Not All Features Matter: Enhancing Few-shot CLIP with Adaptive Prior Refinement](https://mlsanity.com/paper/2304.01195/) (2023)
- [Deep Comprehensive Correlation Mining for Image Clustering](https://mlsanity.com/paper/1904.06925/) (2019)
- [CHiLS: Zero-Shot Image Classification with Hierarchical Label Sets](https://mlsanity.com/paper/2302.02551/) (2023)
- [Clustering-friendly Representation Learning via Instance Discrimination and Feature Decorrelation](https://mlsanity.com/paper/2106.00131/) (2021)
- [Graph Contrastive Clustering](https://mlsanity.com/paper/2104.01429/) (2021)
- [Twin Contrastive Learning for Online Clustering](https://mlsanity.com/paper/2210.11680/) (2022)
- [Improving Zero-shot Generalization and Robustness of Multi-modal Models](https://mlsanity.com/paper/2212.01758/) (2022)
- [MiCE: Mixture of Contrastive Experts for Unsupervised Image Clustering](https://mlsanity.com/paper/2105.01899/) (2021)
- [Enhancing Zero-Shot Vision Models by Label-Free Prompt Distribution Learning and Bias Correcting](https://mlsanity.com/paper/2410.19294/) (2024)
- [Stable Cluster Discrimination for Deep Clustering](https://mlsanity.com/paper/2311.14310/) (2023)
- [Selective Vision-Language Subspace Projection for Few-shot CLIP](https://mlsanity.com/paper/2407.16977/) (2024)
- [Accelerating Diffusion Transformer via Error-Optimized Cache](https://mlsanity.com/paper/2501.19243/) (2025)

## Built on by (most cited first, 5 of 5)

- [Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigation](https://mlsanity.com/paper/2602.24041/) (2026)
- [GuardAlign: Test-time Safety Alignment in Multimodal Large Language Models](https://mlsanity.com/paper/2602.24027/) (2026)
- [CLUBench: A Clustering Benchmark](https://mlsanity.com/paper/2605.29933/) (2026)
- [Reducing Bias and Variance: Generative Semantic Guidance and Bi-Layer Ensemble for Image Clustering](https://mlsanity.com/paper/2605.12961/) (2026)
- [Learning Deep Modality-Shared Self-Expressiveness for Image Clustering with Textual Information](https://mlsanity.com/paper/2608.08418/) (2026)

## Abstract (arXiv)

Image clustering is a classic problem in computer vision, which categorizes images into different groups. Recent studies utilize nouns as external semantic knowledge to improve clustering performance. However, these methods often overlook the inherent ambiguity of nouns, which can distort semantic representations and degrade clustering quality. To address this issue, we propose a hierarChical semAntic alignmEnt method for image clustering, dubbed CAE, which improves clustering performance in a training-free manner. In our approach, we incorporate two complementary types of textual semantics: caption-level descriptions, which convey fine-grained attributes of image content, and noun-level concepts, which represent high-level object categories. We first select relevant nouns from WordNet and descriptions from caption datasets to construct a semantic space aligned with image features. Then, we align image features with selected nouns and captions via optimal transport to obtain a more discriminative semantic space. Finally, we combine the enhanced semantic and image features to perform clustering. Extensive experiments across 8 datasets demonstrate the effectiveness of our method, notably surpassing the state-of-the-art training-free approach with a 4.2% improvement in accuracy and a 2.9% improvement in adjusted rand index (ARI) on the ImageNet-1K dataset.