# Text Clustering with Large Language Model Embeddings

arXiv: https://arxiv.org/abs/2403.15112 · HTML: https://mlsanity.com/paper/2403.15112/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 46 references that are in the map.
- First posted on arXiv 22 Mar 2024.

## Builds on (most cited first, 12 of 12)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [Unsupervised Deep Embedding for Clustering Analysis](https://mlsanity.com/paper/1511.06335/) (2015)
- [A Comprehensive Overview of Large Language Models](https://mlsanity.com/paper/2307.06435/) (2023)
- [The Falcon Series of Open Language Models](https://mlsanity.com/paper/2311.16867/) (2023)
- [Leveraging BERT for Extractive Text Summarization on Lectures](https://mlsanity.com/paper/1906.04165/) (2019)
- [Integrating Semantic Knowledge to Tackle Zero-shot Text Classification](https://mlsanity.com/paper/1903.12626/) (2019)
- [MN-DS: A Multilabeled News Dataset for News Articles Hierarchical Classification](https://mlsanity.com/paper/2212.12061/) (2022)
- [Short Text Clustering with Transformers](https://mlsanity.com/paper/2102.00541/) (2021)

## Built on by (most cited first, 27 of 27)

- [LLMs are Also Effective Embedding Models: An In-depth Overview](https://mlsanity.com/paper/2412.12591/) (2024)
- [Recent advances in text embedding: A Comprehensive Review of Top-Performing Methods on the MTEB Benchmark](https://mlsanity.com/paper/2406.01607/) (2024)
- [TravelAgent: Generative Agents in the Built Environment](https://mlsanity.com/paper/2412.18985/) (2024)
- [Efficient Latent Semantic Clustering for Scaling Test-Time Computation of LLMs](https://mlsanity.com/paper/2506.00344/) (2025)
- [From Topology to Retrieval: Decoding Embedding Spaces with Unified Signatures](https://mlsanity.com/paper/2511.22150/) (2025)
- [Summaries as Centroids for Interpretable and Scalable Text Clustering](https://mlsanity.com/paper/2502.09667/) (2025)
- [Cequel: Cost-Effective Querying of Large Language Models for Text Clustering](https://mlsanity.com/paper/2504.15640/) (2025)
- [Unsupervised Document and Template Clustering using Multimodal Embeddings](https://mlsanity.com/paper/2506.12116/) (2025)
- [HERCULES: Hierarchical Embedding-based Recursive Clustering Using LLMs for Efficient Summarization](https://mlsanity.com/paper/2506.19992/) (2025)
- [t-SNE Exaggerates Clusters, Provably](https://mlsanity.com/paper/2510.07746/) (2025)
- [Evaluating NLP Embedding Models for Handling Science-Specific Symbolic Expressions in Student Texts](https://mlsanity.com/paper/2505.17950/) (2025)
- [Interpreting LLM-as-a-Judge Policies via Verifiable Global Explanations](https://mlsanity.com/paper/2510.08120/) (2025)
- [FlashHead: Efficient Drop-In Replacement for the Classification Head in Language Model Inference](https://mlsanity.com/paper/2603.14591/) (2026)
- [Large Language Model Enhanced Clustering for News Event Detection](https://mlsanity.com/paper/2406.10552/) (2024)
- [wav2graph: A Framework for Supervised Learning Knowledge Graph from Speech](https://mlsanity.com/paper/2408.04174/) (2024)
- [Capturing research literature attitude towards Sustainable Development Goals: an LLM-based topic modeling approach](https://mlsanity.com/paper/2411.02943/) (2024)
- [Explainable Search and Discovery of Visual Cultural Heritage Collections with Multimodal Large Language Models](https://mlsanity.com/paper/2411.04663/) (2024)
- [ANCHOR: Abductive Network Construction with Hierarchical Orchestration for Reliable Probability Inference in Large Language Models](https://mlsanity.com/paper/2605.10328/) (2026)
- [ERASMO: Leveraging Large Language Models for Enhanced Clustering Segmentation](https://mlsanity.com/paper/2410.03738/) (2024)
- [A Dynamic Framework for Semantic Grouping of Common Data Elements (CDE) Using Embeddings and Clustering](https://mlsanity.com/paper/2506.02160/) (2025)
- [Curating art exhibitions using machine learning](https://mlsanity.com/paper/2506.19813/) (2025)
- [An Auditable Pipeline for Fuzzy Full-Text Screening in Systematic Reviews: Integrating Contrastive Semantic Highlighting and LLM Judgment](https://mlsanity.com/paper/2508.15822/) (2025)
- [Chitchat with AI: Understand the supply chain carbon disclosure of companies worldwide through Large Language Model](https://mlsanity.com/paper/2511.00024/) (2025)
- [Mapping the Web of Science, a large-scale graph and text-based dataset with LLM embeddings](https://mlsanity.com/paper/2602.04630/) (2026)
- [Confidence Estimation in Automatic Short Answer Grading with LLMs](https://mlsanity.com/paper/2605.00200/) (2026)
- [Beyond Looking Up, Try Looking Around: Harmonizing Global Structure and Local Consistency in Optimal Transport for Short Text Clustering](https://mlsanity.com/paper/2607.10548/) (2026)
- [A Cascaded Unsupervised-Supervised NLP Pipeline for Detecting Accusatory Language in Public Procurement](https://mlsanity.com/paper/2608.12269/) (2026)

## Abstract (arXiv)

Text clustering is an important method for organising the increasing volume of digital content, aiding in the structuring and discovery of hidden patterns in uncategorised data. The effectiveness of text clustering largely depends on the selection of textual embeddings and clustering algorithms. This study argues that recent advancements in large language models (LLMs) have the potential to enhance this task. The research investigates how different textual embeddings, particularly those utilised in LLMs, and various clustering algorithms influence the clustering of text datasets. A series of experiments were conducted to evaluate the impact of embeddings on clustering results, the role of dimensionality reduction through summarisation, and the adjustment of model size. The findings indicate that LLM embeddings are superior at capturing subtleties in structured language. OpenAI's GPT-3.5 Turbo model yields better results in three out of five clustering metrics across most tested datasets. Most LLM embeddings show improvements in cluster purity and provide a more informative silhouette score, reflecting a refined structural understanding of text data compared to traditional methods. Among the more lightweight models, BERT demonstrates leading performance. Additionally, it was observed that increasing model dimensionality and employing summarisation techniques do not consistently enhance clustering efficiency, suggesting that these strategies require careful consideration for practical application. These results highlight a complex balance between the need for refined text representation and computational feasibility in text clustering applications. This study extends traditional text clustering frameworks by integrating embeddings from LLMs, offering improved methodologies and suggesting new avenues for future research in various types of textual analysis.