mlsanityData to 7 Oct 2026

Distinguish Sense from Nonsense: Out-of-Scope Detection for Virtual Assistants

arXiv 2301.06544 ↗First posted 16 Jan 2023cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Out of Scope (OOS) detection in Conversational AI solutions enables a chatbot to handle a conversation gracefully when it is unable to make sense of the end-user query. Accurately tagging a query as out-of-domain is particularly hard in scenarios when the chatbot is not equipped to handle a topic which has semantic overlap with an existing topic it is trained on. We propose a simple yet effective OOS detection method that outperforms standard OOS detection methods in a real-world deployment of virtual assistants. We discuss the various design and deployment considerations for a cloud platform solution to train virtual assistants and deploy them at scale. Additionally, we propose a collection of datasets that replicates real-world scenarios and show comprehensive results in various settings using both offline and online evaluation metrics.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 15 of its 28 references that are in the map. First posted on arXiv 16 Jan 2023.

Builds on

15 of 28 references.

References · 15

  1. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  2. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,472
  3. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 419
  4. 2017Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samplescited by 389
  5. 2019An Evaluation Dataset for Intent Classification and Out-of-Scope Predictioncited by 378
  6. 2018Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfacescited by 376
  7. 2020Selective Question Answering under Domain Shiftcited by 228
  8. 2019Benchmarking Natural Language Understanding Services for building Conversational Agentscited by 166
  9. 2019Deep Unknown Intent Detection with Margin Losscited by 66
  10. 2021Modeling Discriminative Representations for Out-of-Domain Detection with Supervised Contrastive Learningcited by 42
  11. 2021Out-of-Scope Intent Detection with Self-Supervision and Discriminative Trainingcited by 42
  12. 2019Likelihood Ratios and Generative Classifiers for Unsupervised Out-of-Domain Detection In Task Oriented Dialogcited by 33
3 more
  1. 2021Are Pretrained Transformers Robust in Intent Classification? A Missing Ingredient in Evaluation of Out-of-Scope Intent Detectioncited by 22
  2. 2021Enhancing the Generalization for Intent Classification and Out-of-Domain Detection in SLUcited by 22
  3. 2020HINT3: Raising the bar for Intent Detection in the Wildcited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 1; 2025: 0; 2026: 1.

4 papers.

  1. 2023Chatbots to ChatGPT in a Cybersecurity Space: Evolution, Vulnerabilities, Attacks, Challenges, and Future Recommendationscited by 7
  2. 2023ICL Markup: Structuring In-Context Learning using Soft-Token Tagscited by 5
  3. 2024A new approach for fine-tuning sentence transformers for intent classification and out-of-scope detection taskscited by 4
  4. 2026Building Customer Support AI Agents at 100M-User Scale: An Evaluation-Driven Frameworkcited by 3