Data to 7 Oct 2026

An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction

arXiv 1909.02027 ↗First posted 4 Sep 2019cs.CL cs.AI cs.LGcited by 378

Not on the map. Why?

Abstract (arXiv)

Task-oriented dialog systems need to know when a query falls outside their range of supported intents, but current text classification corpora only define label sets that cover every example. We introduce a new dataset that includes queries that are out-of-scope---i.e., queries that do not fall into any of the system's supported intents. This poses a new challenge because models cannot assume that every query at inference time belongs to a system-supported intent class. Our dataset also covers 150 intent classes over 10 domains, capturing the breadth that a production task-oriented agent must handle. We evaluate a range of benchmark classifiers on our dataset along with several different out-of-scope identification schemes. We find that while the classifiers perform well on in-scope intent classification, they struggle to identify out-of-scope queries. Our dataset and evaluation fill an important gap in the field, offering a way of more rigorously and realistically benchmarking text classification in task-driven dialog systems.

Key facts. Cited by 378 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 49; 2022: 73; 2023: 78; 2024: 69; 2025: 46; 2026: 63.

378 papers.

Sort:

By area:

2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 240
2021PTR: Prompt Tuning with Rules for Text Classificationcited by 238
2025Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactionscited by 236
2021A Simple Fix to Mahalanobis Distance for Improving Near-OOD Detectioncited by 206
2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
2024How to Train Long-Context Language Models (Effectively)cited by 141
2021Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue Systemcited by 139
2021Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Surveycited by 130
2024In-Context Learning with Long-Context Models: An In-Depth Explorationcited by 126
2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 119
2022Plex: Towards Reliability using Pretrained Large Model Extensionscited by 102

38 more

2024Long-Context Language Modeling with Parallel Context Encodingcited by 84
2021Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learningcited by 81
2022Parallel Context Windows for Large Language Modelscited by 81
2023Large Language Models Enable Few-Shot Clusteringcited by 72
2021Contrastive Out-of-Distribution Detection for Pretrained Transformerscited by 70
2023ClusterLLM: Large Language Models as a Guide for Text Clusteringcited by 69
2021Revisiting Mahalanobis Distance for Transformer-Based Out-of-Domain Detectioncited by 67
2024Towards Lifelong Learning of Large Language Models: A Surveycited by 60
2021Neural Data Augmentation via Example Extrapolationcited by 54
2022Data Augmentation for Intent Classification with Off-the-shelf Large Language Modelscited by 49
2022A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awarenesscited by 49
2025Mem-α: Learning Memory Construction via Reinforcement Learningcited by 49
2021Few-Shot Intent Detection via Contrastive Pre-Training and Fine-Tuningcited by 46
2024Recent Advances of Foundation Language Models-based Continual Learning: A Surveycited by 46
2021Modeling Discriminative Representations for Out-of-Domain Detection with Supervised Contrastive Learningcited by 42
2021Out-of-Scope Intent Detection with Self-Supervision and Discriminative Trainingcited by 42
2022MIntRec: A New Dataset for Multimodal Intent Recognitioncited by 41
2023ChatGPT to Replace Crowdsourcing of Paraphrases for Intent Classification: Higher Diversity and Comparable Model Robustnesscited by 41
2022InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuningcited by 39
2021Improved Text Classification via Contrastive Adversarial Trainingcited by 38
2021Stochastic Batch Acquisition: A Simple Baseline for Deep Active Learningcited by 36
2022Nonparametric Uncertainty Quantification for Single Deterministic Neural Networkcited by 36
2023Making LLMs Worth Every Penny: Resource-Limited Text Classification in Bankingcited by 36
2021Few-shot Intent Classification and Slot Filling with Retrieved Examplescited by 35
2021Multilingual and Cross-Lingual Intent Detection from Spoken Datacited by 35
2022Generalized Category Discovery with Decoupled Prototypical Networkcited by 35
2023In-Context Learning for Text Classification with Many Labelscited by 35
2022SPACE-3: Unified Dialog Model Pre-training for Task-Oriented Dialog Understanding and Generationcited by 34
2021Unsupervised Out-of-Domain Detection via Pre-trained Transformerscited by 31
2023A Survey on Out-of-Distribution Detection in NLPcited by 31
2021CINS: Comprehensive Instruction for Few-shot Learning in Task-oriented Dialog Systemscited by 30
2023Task-Optimized Adapters for an End-to-End Task-Oriented Dialogue Systemcited by 30
2023Is Fine-tuning Needed? Pre-trained Language Models Are Near Perfect for Out-of-Domain Detectioncited by 29
2022New Intent Discovery with Pre-training and Contrastive Learningcited by 28
2022SPACE-2: Tree-Structured Semi-Supervised Contrastive Pre-training for Task-Oriented Dialog Understandingcited by 28
2021Effectiveness of Pre-training for Few-shot Intent Classificationcited by 27
2024LLMs Are In-Context Bandit Reinforcement Learnerscited by 27
2024Attention Entropy is a Key Factor: An Analysis of Parallel Context Encoding with Full-attention-based Pre-trained Language Modelscited by 26

50 most cited of 378 All 378 (JSON)