# ESMC: MLLM-Based Embedding Selection for Explainable Multiple Clustering

arXiv: https://arxiv.org/abs/2512.00725 · HTML: https://mlsanity.com/paper/2512.00725/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 47 references that are in the map.
- First posted on arXiv 30 Nov 2025.

## Builds on (most cited first, 19 of 19)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Deep Clustering for Unsupervised Learning of Visual Features](https://mlsanity.com/paper/1807.05520/) (2018)
- [Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding](https://mlsanity.com/paper/2311.16922/) (2023)
- [Eliciting Latent Predictions from Transformers with the Tuned Lens](https://mlsanity.com/paper/2303.08112/) (2023)
- [SCAN: Learning to Classify Images without Labels](https://mlsanity.com/paper/2005.12320/) (2020)
- [Improving Alignment and Robustness with Circuit Breakers](https://mlsanity.com/paper/2406.04313/) (2024)
- [Future Lens: Anticipating Subsequent Tokens from a Single Hidden State](https://mlsanity.com/paper/2311.04897/) (2023)
- [SPICE: Semantic Pseudo-labeling for Image Clustering](https://mlsanity.com/paper/2103.09382/) (2021)
- [Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations](https://mlsanity.com/paper/2410.02762/) (2024)
- [Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revision](https://mlsanity.com/paper/2311.07362/) (2023)
- [Multi-Modal Proxy Learning Towards Personalized Visual Multiple Clustering](https://mlsanity.com/paper/2404.15655/) (2024)
- [Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergence](https://mlsanity.com/paper/2412.13949/) (2024)
- [AugDMC: Data Augmentation Guided Deep Multiple Clustering](https://mlsanity.com/paper/2306.13023/) (2023)
- [Improving Image Clustering With Multiple Pretrained CNN Feature Extractors](https://mlsanity.com/paper/1807.07760/) (2018)
- [Customized Multiple Clustering via Multi-Modal Subspace Proxy Learning](https://mlsanity.com/paper/2411.03978/) (2024)
- [Multiple co-clustering based on nonparametric mixture models with heterogeneous marginal distributions](https://mlsanity.com/paper/1510.06138/) (2015)

## Built on by (most cited first, 1 of 1)

- [Semantic Purification for Conditional Representation Learning](https://mlsanity.com/paper/2602.05464/) (2026)

## Abstract (arXiv)

Typical deep clustering methods, while achieving notable progress, can only provide one clustering result per dataset. This limitation arises from their assumption of a fixed underlying data distribution, which may fail to meet user needs and provide unsatisfactory clustering outcomes. Our work investigates how multi-modal large language models (MLLMs) can be leveraged to achieve user-driven clustering, emphasizing their adaptability to user-specified semantic requirements. However, directly using MLLM output for clustering has risks for producing unstructured and generic image descriptions instead of feature-specific and concrete ones. To address these issues, our method first discovers that MLLMs' hidden states of text tokens are strongly related to the corresponding features, and leverages these embeddings to perform clusterings from any user-defined criteria. We also employ a lightweight clustering head augmented with pseudo-label learning, significantly enhancing clustering accuracy. Extensive experiments demonstrate its competitive performance on diverse datasets and metrics.