# Dial-In LLM: Human-Aligned LLM-in-the-loop Intent Clustering for Customer Service Dialogues

arXiv: https://arxiv.org/abs/2412.09049 · HTML: https://mlsanity.com/paper/2412.09049/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 69 references that are in the map.
- First posted on arXiv 12 Dec 2024.

## Builds on (most cited first, 24 of 24)

- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [C-Pack: Packed Resources For General Chinese Embeddings](https://mlsanity.com/paper/2309.07597/) (2023)
- [LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression](https://mlsanity.com/paper/2310.06839/) (2023)
- [Efficient Intent Detection with Dual Sentence Encoders](https://mlsanity.com/paper/2003.04807/) (2020)
- [An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction](https://mlsanity.com/paper/1909.02027/) (2019)
- [Humans or LLMs as the Judge? A Study on Judgement Biases](https://mlsanity.com/paper/2402.10669/) (2024)
- [Large Language Models Enable Few-Shot Clustering](https://mlsanity.com/paper/2307.00524/) (2023)
- [ClusterLLM: Large Language Models as a Guide for Text Clustering](https://mlsanity.com/paper/2305.14871/) (2023)
- [Discovering New Intents with Deep Aligned Clustering](https://mlsanity.com/paper/2012.08987/) (2020)
- [Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracy](https://mlsanity.com/paper/2402.19379/) (2024)
- [Discovering New Intents via Constrained Deep Adaptive Clustering with Cluster Refinement](https://mlsanity.com/paper/1911.08891/) (2019)
- [Analyzing and Characterizing User Intent in Information-seeking Conversations](https://mlsanity.com/paper/1804.08759/) (2018)
- [IDAS: Intent Discovery with Abstractive Summarization](https://mlsanity.com/paper/2305.19783/) (2023)
- [End-to-end Task-oriented Dialogue: A Survey of Tasks, Methods, and Future Directions](https://mlsanity.com/paper/2311.09008/) (2023)
- [Large Language Models Meet Open-World Intent Discovery and Recognition: An Evaluation of ChatGPT](https://mlsanity.com/paper/2310.10176/) (2023)
- [Other Roles Matter! Enhancing Role-Oriented Dialogue Summarization via Role Interactions](https://mlsanity.com/paper/2205.13190/) (2022)
- [A Brief Survey of Text Mining: Classification, Clustering and Extraction Techniques](https://mlsanity.com/paper/1707.02919/) (2017)
- [Intent Induction from Conversations for Task-Oriented Dialogue Track at DSTC 11](https://mlsanity.com/paper/2304.12982/) (2023)
- [OpenEval: Benchmarking Chinese LLMs across Capability, Alignment and Safety](https://mlsanity.com/paper/2403.12316/) (2024)
- [Familia: A Configurable Topic Modeling Framework for Industrial Text Engineering](https://mlsanity.com/paper/1808.03733/) (2018)
- [Normalized mutual information is a biased measure for classification and community detection](https://mlsanity.com/paper/2307.01282/) (2023)
- [Conformal Intent Classification and Clarification for Fast and Accurate Intent Recognition](https://mlsanity.com/paper/2403.18973/) (2024)
- [Representation Learning for Short Text Clustering](https://mlsanity.com/paper/2109.09894/) (2021)
- [Neural-Bayesian Program Learning for Few-shot Dialogue Intent Parsing](https://mlsanity.com/paper/2410.06190/) (2024)

## Built on by (most cited first, 11 of 11)

- [CompliBench: Benchmarking LLM Judges for Compliance Violation Detection in Dialogue Systems](https://mlsanity.com/paper/2604.12312/) (2026)
- [QualBench: Benchmarking Chinese LLMs with Localized Professional Qualifications for Vertical Domain Evaluation](https://mlsanity.com/paper/2505.05225/) (2025)
- [Improving Language Models with Intentional Analysis](https://mlsanity.com/paper/2502.04689/) (2025)
- [RAL2M: Retrieval Augmented Learning-To-Match Against Hallucination in Compliance-Guaranteed Service Systems](https://mlsanity.com/paper/2601.02917/) (2026)
- [Contextualized Token Discrimination for Speech Search Query Correction](https://mlsanity.com/paper/2509.04393/) (2025)
- [How DDAIR you? Disambiguated Data Augmentation for Intent Recognition](https://mlsanity.com/paper/2601.11234/) (2026)
- [Towards Proactive Information Probing: Customer Service Chatbots Harvesting Value from Conversation](https://mlsanity.com/paper/2604.11077/) (2026)
- [Semantic-Augmented Latent Topic Modeling with LLM-in-the-Loop](https://mlsanity.com/paper/2507.08498/) (2025)
- [Proactive Dialogue Model with Intent Prediction](https://mlsanity.com/paper/2604.27379/) (2026)
- [From Automation to Collaboration: Human-in-the-Loop Methods for Safe and Trustworthy NLP](https://mlsanity.com/paper/2605.25226/) (2026)
- [RAILS: Retrieval-Augmented Incremental LLM Clustering at Scale](https://mlsanity.com/paper/2609.24464/) (2026)

## Abstract (arXiv)

Discovering customer intentions is crucial for automated service agents, yet existing intent clustering methods often fall short due to their reliance on embedding distance metrics and neglect of underlying semantic structures. To address these limitations, we propose an LLM-in-the-loop (LLM-ITL) intent clustering framework, integrating the language understanding capabilities of LLMs into conventional clustering algorithms. Specifically, this paper (1) examines the effectiveness of fine-tuned LLMs in semantic coherence evaluation and intent cluster naming, achieving over 95% accuracy aligned with human judgments; (2) designs an LLM-ITL framework that facilitates the iterative discovery of coherent intent clusters and the optimal number of clusters; and (3) introduces context-aware techniques tailored for customer service dialogue. Since existing English benchmarks lack sufficient semantic diversity and intent coverage, we further present a comprehensive Chinese dialogue intent dataset comprising over 100k real customer service calls with 1,507 human-annotated clusters. The proposed approaches significantly outperform LLM-guided baselines, achieving notable improvements in clustering quality, cost efficiency, and downstream applications. Combined with several best practices, our findings highlight the prominence of LLM-in-the-loop techniques for scalable dialogue data mining.