mlsanityData to 7 Oct 2026

Delving into Spectral Clustering with Vision-Language Representations

arXiv 2602.09586 ↗First posted 10 Feb 2026cs.CVcited by 2

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Spectral clustering is known as a powerful technique in unsupervised data analysis. The vast majority of approaches to spectral clustering are driven by a single modality, leaving the rich information in multi-modal representations untapped. Inspired by the recent success of vision-language pre-training, this paper enriches the landscape of spectral clustering from a single-modal to a multi-modal regime. Particularly, we propose Neural Tangent Kernel Spectral Clustering that leverages cross-modal alignment in pre-trained vision-language models. By anchoring the neural tangent kernel with positive nouns, i.e., those semantically close to the images of interest, we arrive at formulating the affinity between images as a coupling of their visual proximity and semantic overlap. We show that this formulation amplifies within-cluster connections while suppressing spurious ones across clusters, hence encouraging block-diagonal structures. In addition, we present a regularized affinity diffusion mechanism that adaptively ensembles affinity matrices induced by different prompts. Extensive experiments on 16 benchmarks -- including classical, large-scale, fine-grained and domain-shifted datasets -- manifest that our method consistently outperforms the state-of-the-art by a large margin.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 52 of its 100 references that are in the map. First posted on arXiv 10 Feb 2026.

Builds on

52 of 100 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021FILIP: Fine-grained Interactive Language-Image Pre-Training9 Nov

Other references · 51

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  5. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  6. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  7. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
  8. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  9. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  10. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  11. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  12. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
38 more
  1. 2015Unsupervised Deep Embedding for Clustering Analysiscited by 763
  2. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  3. 2018Do Better ImageNet Models Transfer Better?cited by 560
  4. 2018Invariant Information Clustering for Unsupervised Image Classification and Segmentationcited by 454
  5. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 429
  6. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  7. 2020SCAN: Learning to Classify Images without Labelscited by 351
  8. 2020Contrastive Clusteringcited by 264
  9. 2016Variational Deep Embedding: An Unsupervised and Generative Approach to Clusteringcited by 219
  10. 2015Coordinate Descent Algorithmscited by 218
  11. 2017Learning Discrete Representations via Information Maximizing Self-Augmented Trainingcited by 149
  12. 2017Deep Clustering via Joint Convolutional Autoencoder Embedding and Relative Entropy Minimizationcited by 113
  13. 2021SPICE: Semantic Pseudo-labeling for Image Clusteringcited by 104
  14. 2018SpectralNet: Spectral Clustering using Deep Neural Networkscited by 94
  15. 2019Deep Comprehensive Correlation Mining for Image Clusteringcited by 80
  16. 2018ClusterGAN : Latent Space Clustering in Generative Adversarial Networkscited by 75
  17. 2019Deep Spectral Clustering using Dual Autoencoder Networkcited by 71
  18. 2021Clustering-friendly Representation Learning via Instance Discrimination and Feature Decorrelationcited by 65
  19. 2021Graph Contrastive Clusteringcited by 61
  20. 2022Twin Contrastive Learning for Online Clusteringcited by 60
  21. 2021Learning Representation for Clustering via Prototype Scattering and Positive Samplingcited by 58
  22. 2021MiCE: Mixture of Contrastive Experts for Unsupervised Image Clusteringcited by 42
  23. 2021You Never Cluster Alonecited by 38
  24. 2023Image Clustering with External Guidancecited by 36
  25. 2021KNAS: Green Neural Architecture Searchcited by 33
  26. 2023Understanding Transferable Representation Learning and Zero-shot Transfer in CLIPcited by 26
  27. 2022Semantic-Enhanced Image Clusteringcited by 25
  28. 2021Learning a Self-Expressive Network for Subspace Clusteringcited by 24
  29. 2023Stable Cluster Discrimination for Deep Clusteringcited by 24
  30. 2023Decoupled Contrastive Multi-View Clustering with High-Order Random Walkscited by 23
  31. 2022Neural Tangent Kernel: A Surveycited by 22
  32. 2024ConjNorm: Tractable Density Estimation for Out-of-Distribution Detectioncited by 22
  33. 2023DivClust: Controlling Diversity in Deep Clusteringcited by 18
  34. 2021Deep Embedded K-Means Clusteringcited by 12
  35. 2019Spectral Clustering via Ensemble Deep Autoencoder Learning (SC-EDAE)cited by 10
  36. 2025On the Provable Importance of Gradients for Language-Assisted Image Clusteringcited by 8
  37. 2026Explainable LLM Unlearning Through Reasoningcited by 7
  38. 2024Deep Spectral Clustering via Joint Spectral Embedding and Kmeanscited by 4

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Respecting Modality Gap in Post-hoc Out-of-distribution Detection with Pre-trained Vision-Language Modelscited by 0
  2. 2026VLM4Cluster: Benchmarking Deep Clustering In the Era of Vision-Language Pre-trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Language-Assisted Image Clustering Guided by Discriminative Relational Signals and Adaptive Semantic Centers
  2. 2026Learning Deep Modality-Shared Self-Expressiveness for Image Clustering with Textual Information
  3. 2026KODA: Contrastive Representation Comparison and Alignment for Vision-Language Foundation Models
  4. 2022Perceptual Grouping in Contrastive Vision-Language Models
  5. 2026Beyond Compression: Quantifying Spectral Accessibility in Vision Representations
  6. 2026Structure and Redundancy in Large Language Models: A Spectral Study via Random Matrix Theory
  7. 2026Your CLIP has 164 dimensions of noise: Exploring the embeddings covariance eigenspectrum of contrastively pretrained vision-language transformers
  8. 2024Semantic Equitable Clustering: A Simple and Effective Strategy for Clustering Vision Tokens