# Intent Mining from past conversations for conversational agent

arXiv: https://arxiv.org/abs/2005.11014 · HTML: https://mlsanity.com/paper/2005.11014/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 May 2020.

## Built on by (most cited first, 25 of 25)

- [New Intent Discovery with Pre-training and Contrastive Learning](https://mlsanity.com/paper/2205.12914/) (2022)
- [Intent Induction from Conversations for Task-Oriented Dialogue Track at DSTC 11](https://mlsanity.com/paper/2304.12982/) (2023)
- [Open Intent Discovery through Unsupervised Semantic Clustering and Dependency Parsing](https://mlsanity.com/paper/2104.12114/) (2021)
- [NatCS: Eliciting Natural Customer Support Dialogues](https://mlsanity.com/paper/2305.03007/) (2023)
- [ProtoInfoMax: Prototypical Networks with Mutual Information Maximization for Out-of-Domain Detection](https://mlsanity.com/paper/2108.12229/) (2021)
- [Unsupervised Extraction of Dialogue Policies from Conversations](https://mlsanity.com/paper/2406.15214/) (2024)
- [LANID: LLM-assisted New Intent Discovery](https://mlsanity.com/paper/2503.23740/) (2025)
- [Dialog Intent Induction via Density-based Deep Clustering Ensemble](https://mlsanity.com/paper/2201.06731/) (2022)
- [Confidence Calibration for Intent Detection via Hyperspherical Space and Rebalanced Accuracy-Uncertainty Loss](https://mlsanity.com/paper/2203.09278/) (2022)
- [Toward Open-domain Slot Filling via Self-supervised Co-training](https://mlsanity.com/paper/2303.13801/) (2023)
- [Training LLMs with Reinforcement Learning for Intent-Aware Personalized Question Answering](https://mlsanity.com/paper/2605.12645/) (2026)
- [Convex Polytope Modelling for Unsupervised Derivation of Semantic Structure for Data-efficient Natural Language Understanding](https://mlsanity.com/paper/2201.10588/) (2022)
- [Adapting Task-Oriented Dialogue Models for Email Conversations](https://mlsanity.com/paper/2208.09439/) (2022)
- [Building the Intent Landscape of Real-World Conversational Corpora with Extractive Question-Answering Transformers](https://mlsanity.com/paper/2208.12886/) (2022)
- [Analysis of Utterance Embeddings and Clustering Methods Related to Intent Induction for Task-Oriented Dialogue](https://mlsanity.com/paper/2212.02021/) (2022)
- [Multi-Stage Coarse-to-Fine Contrastive Learning for Conversation Intent Induction](https://mlsanity.com/paper/2303.05034/) (2023)
- [Towards a Fully Unsupervised Framework for Intent Induction in Customer Support Dialogues](https://mlsanity.com/paper/2307.15410/) (2023)
- [Controllable Discovery of Intents: Incremental Deep Clustering Using Semi-Supervised Contrastive Learning](https://mlsanity.com/paper/2410.14755/) (2024)
- [A Risk Taxonomy and Reflection Tool for Large Language Model Adoption in Public Health](https://mlsanity.com/paper/2411.02594/) (2024)
- [DORIC : Domain Robust Fine-Tuning for Open Intent Clustering through Dependency Parsing](https://mlsanity.com/paper/2303.09827/) (2023)
- [Multi-View Zero-Shot Open Intent Induction from Dialogues: Multi Domain Batch and Proxy Gradient Transfer](https://mlsanity.com/paper/2303.13099/) (2023)
- [Algorithms for automatic intents extraction and utterances classification for goal-oriented dialogue systems](https://mlsanity.com/paper/2312.09658/) (2023)
- [One Stone, Four Birds: A Comprehensive Solution for QA System Using Supervised Contrastive Learning](https://mlsanity.com/paper/2407.09011/) (2024)
- [Pseudo-Label Enhanced Prototypical Contrastive Learning for Uniformed Intent Discovery](https://mlsanity.com/paper/2410.20219/) (2024)
- [CATCH: A Controllable Theme Detection Framework with Contextualized Clustering and Hierarchical Generation](https://mlsanity.com/paper/2512.21715/) (2025)

## Abstract (arXiv)

Conversational systems are of primary interest in the AI community. Chatbots are increasingly being deployed to provide round-the-clock support and to increase customer engagement. Many of the commercial bot building frameworks follow a standard approach that requires one to build and train an intent model to recognize a user input. Intent models are trained in a supervised setting with a collection of textual utterance and intent label pairs. Gathering a substantial and wide coverage of training data for different intent is a bottleneck in the bot building process. Moreover, the cost of labeling a hundred to thousands of conversations with intent is a time consuming and laborious job. In this paper, we present an intent discovery framework that involves 4 primary steps: Extraction of textual utterances from a conversation using a pre-trained domain agnostic Dialog Act Classifier (Data Extraction), automatic clustering of similar user utterances (Clustering), manual annotation of clusters with an intent label (Labeling) and propagation of intent labels to the utterances from the previous step, which are not mapped to any cluster (Label Propagation); to generate intent training data from raw conversations. We have introduced a novel density-based clustering algorithm ITER-DBSCAN for unbalanced data clustering. Subject Matter Expert (Annotators with domain expertise) manually looks into the clustered user utterances and provides an intent label for discovery. We conducted user studies to validate the effectiveness of the trained intent model generated in terms of coverage of intents, accuracy and time saving concerning manual annotation. Although the system is developed for building an intent model for the conversational system, this framework can also be used for a short text clustering or as a labeling framework.