# Context-Aware Clustering using Large Language Models

arXiv: https://arxiv.org/abs/2405.00988 · HTML: https://mlsanity.com/paper/2405.00988/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 37 references that are in the map.
- First posted on arXiv 2 May 2024.

## Builds on (most cited first, 15 of 15)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [BERTopic: Neural topic modeling with a class-based TF-IDF procedure](https://mlsanity.com/paper/2203.05794/) (2022)
- [Text Is All You Need: Learning Language Representations for Sequential Recommendation](https://mlsanity.com/paper/2305.13731/) (2023)
- [Supporting Clustering with Contrastive Learning](https://mlsanity.com/paper/2103.12953/) (2021)
- [Large Language Models Enable Few-Shot Clustering](https://mlsanity.com/paper/2307.00524/) (2023)
- [ClusterLLM: Large Language Models as a Guide for Text Clustering](https://mlsanity.com/paper/2305.14871/) (2023)
- [Discovering New Intents with Deep Aligned Clustering](https://mlsanity.com/paper/2012.08987/) (2020)
- [Discovering New Intents via Constrained Deep Adaptive Clustering with Cluster Refinement](https://mlsanity.com/paper/1911.08891/) (2019)
- [Generalized Category Discovery with Decoupled Prototypical Network](https://mlsanity.com/paper/2211.15115/) (2022)
- [Topic Discovery via Latent Space Clustering of Pretrained Language Model Representations](https://mlsanity.com/paper/2202.04582/) (2022)
- [New Intent Discovery with Pre-training and Contrastive Learning](https://mlsanity.com/paper/2205.12914/) (2022)

## Built on by (most cited first, 9 of 9)

- [Tool-Planner: Task Planning with Clusters across Multiple Tools](https://mlsanity.com/paper/2406.03807/) (2024)
- [Organizing Unstructured Image Collections using Natural Language](https://mlsanity.com/paper/2410.05217/) (2024)
- [Summaries as Centroids for Interpretable and Scalable Text Clustering](https://mlsanity.com/paper/2502.09667/) (2025)
- [HERCULES: Hierarchical Embedding-based Recursive Clustering Using LLMs for Efficient Summarization](https://mlsanity.com/paper/2506.19992/) (2025)
- [Cluster-R1: Large Reasoning Models Are Instruction-following Clustering Agents](https://mlsanity.com/paper/2603.23518/) (2026)
- [RWKV-edge: Deeply Compressed RWKV for Resource-Constrained Devices](https://mlsanity.com/paper/2412.10856/) (2024)
- [ERASMO: Leveraging Large Language Models for Enhanced Clustering Segmentation](https://mlsanity.com/paper/2410.03738/) (2024)
- [In-Context Clustering with Large Language Models](https://mlsanity.com/paper/2510.08466/) (2025)
- [AudioLens: Multi-Perspective Speech Clustering with Reasoning Audio-Language Models](https://mlsanity.com/paper/2608.25177/) (2026)

## Abstract (arXiv)

Despite the remarkable success of Large Language Models (LLMs) in text understanding and generation, their potential for text clustering tasks remains underexplored. We observed that powerful closed-source LLMs provide good quality clusterings of entity sets but are not scalable due to the massive compute power required and the associated costs. Thus, we propose CACTUS (Context-Aware ClusTering with aUgmented triplet losS), a systematic approach that leverages open-source LLMs for efficient and effective supervised clustering of entity subsets, particularly focusing on text-based entities. Existing text clustering methods fail to effectively capture the context provided by the entity subset. Moreover, though there are several language modeling based approaches for clustering, very few are designed for the task of supervised clustering. This paper introduces a novel approach towards clustering entity subsets using LLMs by capturing context via a scalable inter-entity attention mechanism. We propose a novel augmented triplet loss function tailored for supervised clustering, which addresses the inherent challenges of directly applying the triplet loss to this problem. Furthermore, we introduce a self-supervised clustering task based on text augmentation techniques to improve the generalization of our model. For evaluation, we collect ground truth clusterings from a closed-source LLM and transfer this knowledge to an open-source LLM under the supervised clustering framework, allowing a faster and cheaper open-source model to perform the same task. Experiments on various e-commerce query and product clustering datasets demonstrate that our proposed approach significantly outperforms existing unsupervised and supervised baselines under various external clustering evaluation metrics.