# Towards Open Intent Discovery for Conversational Text

arXiv: https://arxiv.org/abs/1904.08524 · HTML: https://mlsanity.com/paper/1904.08524/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Apr 2019.

## Built on by (most cited first, 13 of 13)

- [A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awareness](https://mlsanity.com/paper/2205.00403/) (2022)
- [Ask what's missing and what's useful: Improving Clarification Question Generation using Global Knowledge](https://mlsanity.com/paper/2104.06828/) (2021)
- [Open-world Machine Learning: Applications, Challenges, and Opportunities](https://mlsanity.com/paper/2105.13448/) (2021)
- [Cross-Data Knowledge Graph Construction for LLM-enabled Educational Question-Answering System: A Case Study at HCMUT](https://mlsanity.com/paper/2404.09296/) (2024)
- [Open Intent Discovery through Unsupervised Semantic Clustering and Dependency Parsing](https://mlsanity.com/paper/2104.12114/) (2021)
- [Z-BERT-A: a zero-shot Pipeline for Unknown Intent detection](https://mlsanity.com/paper/2208.07084/) (2022)
- [Discriminant Distance-Aware Representation on Deterministic Uncertainty Quantification Methods](https://mlsanity.com/paper/2402.12664/) (2024)
- [Integration of Old and New Knowledge for Generalized Intent Discovery: A Consistency-driven Prototype-Prompting Framework](https://mlsanity.com/paper/2506.08490/) (2025)
- [Training LLMs with Reinforcement Learning for Intent-Aware Personalized Question Answering](https://mlsanity.com/paper/2605.12645/) (2026)
- [Building the Intent Landscape of Real-World Conversational Corpora with Extractive Question-Answering Transformers](https://mlsanity.com/paper/2208.12886/) (2022)
- [Pseudo-Label Enhanced Prototypical Contrastive Learning for Uniformed Intent Discovery](https://mlsanity.com/paper/2410.20219/) (2024)
- [OpenCML: End-to-End Framework of Open-world Machine Learning to Learn Unknown Classes Incrementally](https://mlsanity.com/paper/2511.19491/) (2025)
- [IntPro: A Proxy Agent for Context-Aware Intent Understanding via Retrieval-conditioned Inference](https://mlsanity.com/paper/2603.03325/) (2026)

## Abstract (arXiv)

Detecting and identifying user intent from text, both written and spoken, plays an important role in modelling and understand dialogs. Existing research for intent discovery model it as a classification task with a predefined set of known categories. To generailze beyond these preexisting classes, we define a new task of textitopen intent discovery. We investigate how intent can be generalized to those not seen during training. To this end, we propose a two-stage approach to this task - predicting whether an utterance contains an intent, and then tagging the intent in the input utterance. Our model consists of a bidirectional LSTM with a CRF on top to capture contextual semantics, subject to some constraints. Self-attention is used to learn long distance dependencies. Further, we adapt an adversarial training approach to improve robustness and perforamce across domains. We also present a dataset of 25k real-life utterances that have been labelled via crowd sourcing. Our experiments across different domains and real-world datasets show the effectiveness of our approach, with less than 100 annotated examples needed per unique domain to recognize diverse intents. The approach outperforms state-of-the-art baselines by 5-15% F1 score points.