mlsanityData to 7 Oct 2026

Are Pretrained Transformers Robust in Intent Classification? A Missing Ingredient in Evaluation of Out-of-Scope Intent Detection

arXiv 2106.04564 ↗First posted 8 Jun 2021cs.CL cs.AIcited by 22

Not on the map. Why?

Abstract (arXiv)

Pre-trained Transformer-based models were reported to be robust in intent classification. In this work, we first point out the importance of in-domain out-of-scope detection in few-shot intent recognition tasks and then illustrate the vulnerability of pre-trained Transformer-based models against samples that are in-domain but out-of-scope (ID-OOS). We construct two new datasets, and empirically show that pre-trained models do not perform well on both ID-OOS examples and general out-of-scope examples, especially on fine-grained few-shot intent detection tasks. To figure out how the models mistakenly classify ID-OOS intents as in-scope intents, we further conduct analysis on confidence scores and the overlapping keywords, as well as point out several prospective directions for future work. Resources are available on https://github.com/jianguoz/Few-Shot-Intent-Detection.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 19 of its 27 references that are in the map. First posted on arXiv 8 Jun 2021.

Builds on

19 of 27 references.

References · 19

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  6. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  7. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 419
  8. 2019An Evaluation Dataset for Intent Classification and Out-of-Scope Predictioncited by 378
  9. 2020Pretrained Transformers Improve Out-of-Distribution Robustnesscited by 293
  10. 2019Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systemscited by 215
  11. 2020TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialoguecited by 185
  12. 2018Neural Approaches to Conversational AIcited by 164
7 more
  1. 2020Unsupervised Domain Clusters in Pretrained Language Modelscited by 160
  2. 2017DOC: Deep Open Classification of Text Documentscited by 109
  3. 2021Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoderscited by 88
  4. 2019Find or Classify? Dual Strategy for Slot-Value Predictions on Multi-Domain Dialog State Trackingcited by 75
  5. 2019Out-of-domain Detection for Natural Language Understanding in Dialog Systemscited by 62
  6. 2020Discriminative Nearest Neighbor Few-Shot Intent Detection by Transferring Natural Language Inferencecited by 47
  7. 2021Unsupervised Out-of-Domain Detection via Pre-trained Transformerscited by 31

Built on by

Cited by year

Citing papers per year: 2021: 4; 2022: 3; 2023: 7; 2024: 8.

22 papers.

Sort:

By area:

  1. 2022Plex: Towards Reliability using Pretrained Large Model Extensionscited by 102
  2. 2021Continuous-Time Sequential Recommendation with Temporal Graph Collaborative Transformercited by 69
  3. 2021Few-Shot Intent Detection via Contrastive Pre-Training and Fine-Tuningcited by 46
  4. 2021Modeling Sequences as Distributions with Uncertainty for Sequential Recommendationcited by 22
  5. 2023DialogStudio: Towards Richest and Most Diverse Unified Dataset Collection for Conversational AIcited by 21
  6. 2022Enhancing Out-of-Distribution Detection in Natural Language Understanding via Implicit Layer Ensemblecited by 9
  7. 2024Komodo: A Linguistic Expedition into Indonesia's Regional Languagescited by 9
  8. 2024IntentGPT: Few-shot Intent Discovery with Large Language Modelscited by 8
  9. 2023ICL Markup: Structuring In-Context Learning using Soft-Token Tagscited by 5
  10. 2022Z-BERT-A: a zero-shot Pipeline for Unknown Intent detectioncited by 4
  11. 2023Distinguish Sense from Nonsense: Out-of-Scope Detection for Virtual Assistantscited by 4
  12. 2023Zero-Shot Slot and Intent Detection in Low-Resource Languagescited by 4
10 more
  1. 2023Revealing User Familiarity Bias in Task-Oriented Dialogue via Interactive Evaluationcited by 4
  2. 2024Domain Adaptation in Intent Classification Systems: A Reviewcited by 4
  3. 2023Dense Retrieval as Indirect Supervision for Large-space Decision Makingcited by 3
  4. 2023Probing Out-of-Distribution Robustness of Language Models with Parameter-Efficient Transfer Learningcited by 2
  5. 2024Generating Hard-Negative Out-of-Scope Data with ChatGPT for Intent Classificationcited by 2
  6. 2024From Zero to Hero: Cold-Start Anomaly Detectioncited by 2
  7. 2024DarijaBanking: A New Resource for Overcoming Language Barriers in Banking Intent Detection for Moroccan Arabic Speakerscited by 1
  8. 2024Practical token pruning for foundation models in few-shot conversational virtual assistant systemscited by 1
  9. 2021Exploring the Limits of Natural Language Inference Based Setup for Few-Shot Intent Detectioncited by 0
  10. 2024Diversity-grounded Channel Prototypical Learning for Out-of-Distribution Intent Detectioncited by 0