# Multi-Modal Proxy Learning Towards Personalized Visual Multiple Clustering

arXiv: https://arxiv.org/abs/2404.15655 · HTML: https://mlsanity.com/paper/2404.15655/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 36 references that are in the map.
- First posted on arXiv 24 Apr 2024.

## Builds on (most cited first, 16 of 16)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [CLIP-Adapter: Better Vision-Language Models with Feature Adapters](https://mlsanity.com/paper/2110.04544/) (2021)
- [Unsupervised Deep Embedding for Clustering Analysis](https://mlsanity.com/paper/1511.06335/) (2015)
- [Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models](https://mlsanity.com/paper/2209.07511/) (2022)
- [Visual Classification via Description from Large Language Models](https://mlsanity.com/paper/2210.07183/) (2022)
- [SoftTriple Loss: Deep Metric Learning Without Triplet Sampling](https://mlsanity.com/paper/1909.05235/) (2019)
- [Unsupervised Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2204.03649/) (2022)
- [Unsupervised Visual Representation Learning by Online Constrained K-Means](https://mlsanity.com/paper/2105.11527/) (2021)
- [Intra-Modal Proxy Learning for Zero-Shot Visual Categorization with CLIP](https://mlsanity.com/paper/2310.19752/) (2023)
- [AugDMC: Data Augmentation Guided Deep Multiple Clustering](https://mlsanity.com/paper/2306.13023/) (2023)
- [Improving Image Clustering With Multiple Pretrained CNN Feature Extractors](https://mlsanity.com/paper/1807.07760/) (2018)
- [Multi-View Multiple Clusterings using Deep Matrix Factorization](https://mlsanity.com/paper/1911.11396/) (2019)
- [Improved Visual Fine-tuning with Natural Language Supervision](https://mlsanity.com/paper/2304.01489/) (2023)

## Built on by (most cited first, 21 of 21)

- [Temporal Working Memory: Query-Guided Segment Refinement for Enhanced Multimodal Understanding](https://mlsanity.com/paper/2502.06020/) (2025)
- [UniQA: Unified Vision-Language Pre-training for Image Quality and Aesthetic Assessment](https://mlsanity.com/paper/2406.01069/) (2024)
- [Organizing Unstructured Image Collections using Natural Language](https://mlsanity.com/paper/2410.05217/) (2024)
- [Conditional Representation Learning for Customized Tasks](https://mlsanity.com/paper/2510.04564/) (2025)
- [Music Audio-Visual Question Answering Requires Specialized Multimodal Designs](https://mlsanity.com/paper/2505.20638/) (2025)
- [Learning Sparsity for Effective and Efficient Music Performance Question Answering](https://mlsanity.com/paper/2506.01319/) (2025)
- [FD2Talk: Towards Generalized Talking Head Generation with Facial Decoupled Diffusion Model](https://mlsanity.com/paper/2408.09384/) (2024)
- [Open Ad-hoc Categorization with Contextualized Feature Learning](https://mlsanity.com/paper/2512.16202/) (2025)
- [Text-Guided Alternative Image Clustering](https://mlsanity.com/paper/2406.18589/) (2024)
- [Coarse-to-fine Alignment Makes Better Speech-image Retrieval](https://mlsanity.com/paper/2408.13119/) (2024)
- [Agent-Centric Personalized Multiple Clustering with Multi-Modal LLMs](https://mlsanity.com/paper/2503.22241/) (2025)
- [ESMC: MLLM-Based Embedding Selection for Explainable Multiple Clustering](https://mlsanity.com/paper/2512.00725/) (2025)
- [Towards Spoken Language Understanding via Multi-level Multi-grained Contrastive Learning](https://mlsanity.com/paper/2405.20852/) (2024)
- [Recovering Global Data Distribution Locally in Federated Learning](https://mlsanity.com/paper/2409.14063/) (2024)
- [Interpretable Text-Guided Image Clustering via Iterative Search](https://mlsanity.com/paper/2506.12514/) (2025)
- [Multi-modal Dynamic Proxy Learning for Personalized Multiple Clustering](https://mlsanity.com/paper/2511.07274/) (2025)
- [Semantic Purification for Conditional Representation Learning](https://mlsanity.com/paper/2602.05464/) (2026)
- [Edu-MMBias: A Three-Tier Multimodal Benchmark for Auditing Social Bias in Vision-Language Models under Educational Contexts](https://mlsanity.com/paper/2604.10200/) (2026)
- [Hierarchical Textual Knowledge for Enhanced Image Clustering](https://mlsanity.com/paper/2604.11144/) (2026)
- [Universal Guideline-Driven Image Clustering via a Hybrid LLM Agent](https://mlsanity.com/paper/2606.24094/) (2026)
- [Controlling Embedding Spaces with Text-Conditioned Transformations](https://mlsanity.com/paper/2607.22919/) (2026)

## Abstract (arXiv)

Multiple clustering has gained significant attention in recent years due to its potential to reveal multiple hidden structures of data from different perspectives. The advent of deep multiple clustering techniques has notably advanced the performance by uncovering complex patterns and relationships within large datasets. However, a major challenge arises as users often do not need all the clusterings that algorithms generate, and figuring out the one needed requires a substantial understanding of each clustering result. Traditionally, aligning a user's brief keyword of interest with the corresponding vision components was challenging, but the emergence of multi-modal and large language models (LLMs) has begun to bridge this gap. In response, given unlabeled target visual data, we propose Multi-MaP, a novel method employing a multi-modal proxy learning process. It leverages CLIP encoders to extract coherent text and image embeddings, with GPT-4 integrating users' interests to formulate effective textual contexts. Moreover, reference word constraint and concept-level constraint are designed to learn the optimal text proxy according to the user's interest. Multi-MaP not only adeptly captures a user's interest via a keyword but also facilitates identifying relevant clusterings. Our extensive experiments show that Multi-MaP consistently outperforms state-of-the-art methods in all benchmark multi-clustering vision tasks. Our code is available at https://github.com/Alexander-Yao/Multi-MaP.