# Supporting Clustering with Contrastive Learning

arXiv: https://arxiv.org/abs/2103.12953 · HTML: https://mlsanity.com/paper/2103.12953/

## Key facts

- Cited by 90 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 61 references that are in the map.
- First posted on arXiv 24 Mar 2021.

## Builds on (most cited first, 30 of 35)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [Unsupervised Deep Embedding for Clustering Analysis](https://mlsanity.com/paper/1511.06335/) (2015)
- [Learning Representations by Maximizing Mutual Information Across Views](https://mlsanity.com/paper/1906.00910/) (2019)
- [Prototypical Contrastive Learning of Unsupervised Representations](https://mlsanity.com/paper/2005.04966/) (2020)
- [Hard Negative Mixing for Contrastive Learning](https://mlsanity.com/paper/2010.01028/) (2020)
- [Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networks](https://mlsanity.com/paper/1406.6909/) (2014)
- [DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations](https://mlsanity.com/paper/2006.03659/) (2020)
- [Variational Deep Embedding: An Unsupervised and Generative Approach to Clustering](https://mlsanity.com/paper/1611.05148/) (2016)
- [Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relations](https://mlsanity.com/paper/1805.06201/) (2018)
- [Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases](https://mlsanity.com/paper/2007.13916/) (2020)
- [COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining](https://mlsanity.com/paper/2102.08473/) (2021)
- [SpectralNet: Spectral Clustering using Deep Neural Networks](https://mlsanity.com/paper/1801.01587/) (2018)
- [Mixture Models for Diverse Machine Translation: Tricks of the Trade](https://mlsanity.com/paper/1902.07816/) (2019)
- [CoDA: Contrast-enhanced and Diversity-promoting Data Augmentation for Natural Language Understanding](https://mlsanity.com/paper/2010.08670/) (2020)

## Built on by (most cited first, 30 of 90)

- [TEST: Text Prototype Aligned Embedding to Activate LLM's Ability for Time Series](https://mlsanity.com/paper/2308.08241/) (2023)
- [CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2109.09209/) (2021)
- [A Comprehensive Survey on Deep Clustering: Taxonomy, Challenges, and Future Directions](https://mlsanity.com/paper/2206.07579/) (2022)
- [Large Language Models Enable Few-Shot Clustering](https://mlsanity.com/paper/2307.00524/) (2023)
- [ClusterLLM: Large Language Models as a Guide for Text Clustering](https://mlsanity.com/paper/2305.14871/) (2023)
- [Twin Contrastive Learning for Online Clustering](https://mlsanity.com/paper/2210.11680/) (2022)
- [Detect Rumors in Microblog Posts for Low-Resource Domains via Adversarial Contrastive Learning](https://mlsanity.com/paper/2204.08143/) (2022)
- [You Never Cluster Alone](https://mlsanity.com/paper/2106.01908/) (2021)
- [Mitigating Data Sparsity for Short Text Topic Modeling by Topic-Semantic Contrastive Learning](https://mlsanity.com/paper/2211.12878/) (2022)
- [UCTopic: Unsupervised Contrastive Learning for Phrase Representations and Topic Mining](https://mlsanity.com/paper/2202.13469/) (2022)
- [OpenCon: Open-world Contrastive Learning](https://mlsanity.com/paper/2208.02764/) (2022)
- [Pairwise Supervised Contrastive Learning of Sentence Representations](https://mlsanity.com/paper/2109.05424/) (2021)
- [New Intent Discovery with Pre-training and Contrastive Learning](https://mlsanity.com/paper/2205.12914/) (2022)
- [IDAS: Intent Discovery with Abstractive Summarization](https://mlsanity.com/paper/2305.19783/) (2023)
- [When and How Does Known Class Help Discover Unknown Ones? Provable Understanding Through Spectral Analysis](https://mlsanity.com/paper/2308.05017/) (2023)
- [Self-supervised Contrastive Attributed Graph Clustering](https://mlsanity.com/paper/2110.08264/) (2021)
- [ContraCLM: Contrastive Learning For Causal Language Model](https://mlsanity.com/paper/2210.01185/) (2022)
- [Towards Textual Out-of-Domain Detection without In-Domain Labels](https://mlsanity.com/paper/2203.11396/) (2022)
- [Unsupervised Representation Learning for Time Series: A Review](https://mlsanity.com/paper/2308.01578/) (2023)
- [Debiasing Multimodal Sarcasm Detection with Contrastive Learning](https://mlsanity.com/paper/2312.10493/) (2023)
- [Improving Event Representation via Simultaneous Weakly Supervised Contrastive Learning and Clustering](https://mlsanity.com/paper/2203.07633/) (2022)
- [Beyond Prompting: Making Pre-trained Language Models Better Zero-shot Learners by Clustering Representations](https://mlsanity.com/paper/2210.16637/) (2022)
- [Unsupervised Cross-Domain Rumor Detection with Contrastive Learning and Cross-Attention](https://mlsanity.com/paper/2303.11945/) (2023)
- [Self-supervised Context-aware Style Representation for Expressive Speech Synthesis](https://mlsanity.com/paper/2206.12559/) (2022)
- [Intent Induction from Conversations for Task-Oriented Dialogue Track at DSTC 11](https://mlsanity.com/paper/2304.12982/) (2023)
- [Self-Supervised Learning of Time Series Representation via Diffusion Process and Imputation-Interpolation-Forecasting Mask](https://mlsanity.com/paper/2405.05959/) (2024)
- [Towards Personalized Review Summarization by Modeling Historical Reviews from Customer and Product Separately](https://mlsanity.com/paper/2301.11682/) (2023)
- [A Clustering Framework for Unsupervised and Semi-supervised New Intent Discovery](https://mlsanity.com/paper/2304.07699/) (2023)
- [Context-Aware Clustering using Large Language Models](https://mlsanity.com/paper/2405.00988/) (2024)
- [Explanation Graph Generation via Pre-trained Language Models: An Empirical Study with Contrastive Learning](https://mlsanity.com/paper/2204.04813/) (2022)

## Abstract (arXiv)

Unsupervised clustering aims at discovering the semantic categories of data according to some distance measured in the representation space. However, different categories often overlap with each other in the representation space at the beginning of the learning process, which poses a significant challenge for distance-based clustering in achieving good separation between different categories. To this end, we propose Supporting Clustering with Contrastive Learning (SCCL) -- a novel framework to leverage contrastive learning to promote better separation. We assess the performance of SCCL on short text clustering and show that SCCL significantly advances the state-of-the-art results on most benchmark datasets with 3%-11% improvement on Accuracy and 4%-15% improvement on Normalized Mutual Information. Furthermore, our quantitative analysis demonstrates the effectiveness of SCCL in leveraging the strengths of both bottom-up instance discrimination and top-down clustering to achieve better intra-cluster and inter-cluster distances when evaluated with the ground truth cluster labels.