# In-Context Clustering with Large Language Models

arXiv: https://arxiv.org/abs/2510.08466 · HTML: https://mlsanity.com/paper/2510.08466/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 32 of its 52 references that are in the map.
- First posted on arXiv 9 Oct 2025.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://mlsanity.com/paper/1709.00029/) (2017)
- [What Can Transformers Learn In-Context? A Case Study of Simple Function Classes](https://mlsanity.com/paper/2208.01066/) (2022)
- [Multimodal Few-Shot Learning with Frozen Language Models](https://mlsanity.com/paper/2106.13884/) (2021)
- [An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models](https://mlsanity.com/paper/2403.06764/) (2024)
- [ImageNet-21K Pretraining for the Masses](https://mlsanity.com/paper/2104.10972/) (2021)
- [Towards Reasoning in Large Language Models: A Survey](https://mlsanity.com/paper/2212.10403/) (2022)
- [LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models](https://mlsanity.com/paper/2407.07895/) (2024)
- [Large Language Models Are Zero-Shot Time Series Forecasters](https://mlsanity.com/paper/2310.07820/) (2023)
- [MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention](https://mlsanity.com/paper/2407.02490/) (2024)
- [Long-context LLMs Struggle with Long In-context Learning](https://mlsanity.com/paper/2404.02060/) (2024)
- [Self-Supervised Learning by Cross-Modal Audio-Video Clustering](https://mlsanity.com/paper/1911.12667/) (2019)
- [Using Deep Learning for Image-Based Plant Disease Detection](https://mlsanity.com/paper/1604.03169/) (2016)
- [The emergence of clusters in self-attention dynamics](https://mlsanity.com/paper/2305.05465/) (2023)
- [LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Models](https://mlsanity.com/paper/2404.01230/) (2024)
- [Large Language Models Enable Few-Shot Clustering](https://mlsanity.com/paper/2307.00524/) (2023)
- [ClusterLLM: Large Language Models as a Guide for Text Clustering](https://mlsanity.com/paper/2305.14871/) (2023)
- [From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examples](https://mlsanity.com/paper/2404.07544/) (2024)
- [MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformer](https://mlsanity.com/paper/2403.02991/) (2024)
- [LLM as Dataset Analyst: Subpopulation Structure Discovery with Large Language Model](https://mlsanity.com/paper/2405.02363/) (2024)
- [Image Clustering Conditioned on Text Criteria](https://mlsanity.com/paper/2310.18297/) (2023)
- [Meta Learning to Bridge Vision and Language Models for Multimodal Few-Shot Learning](https://mlsanity.com/paper/2302.14794/) (2023)
- [Improving Image Clustering With Multiple Pretrained CNN Feature Extractors](https://mlsanity.com/paper/1807.07760/) (2018)

## Abstract (arXiv)

We propose In-Context Clustering (ICC), a flexible LLM-based procedure for clustering data from diverse distributions. Unlike traditional clustering algorithms constrained by predefined similarity measures, ICC flexibly captures complex relationships among inputs through an attention mechanism. We show that pretrained LLMs exhibit impressive zero-shot clustering capabilities on text-encoded numeric data, with attention matrices showing salient cluster patterns. Spectral clustering using attention matrices offers surprisingly competitive performance. We further enhance the clustering capabilities of LLMs on numeric and image data through fine-tuning using the Next Token Prediction (NTP) loss. Moreover, the flexibility of LLM prompting enables text-conditioned image clustering, a capability that classical clustering methods lack. Our work extends in-context learning to an unsupervised setting, showcasing the effectiveness and flexibility of LLMs for clustering. Our code is available at https://agenticlearning.ai/icc.