mlsanityData to 7 Oct 2026

Unsupervised Out-of-Domain Detection via Pre-trained Transformers

arXiv 2106.00948 ↗First posted 2 Jun 2021cs.CL cs.LGcited by 31

Not on the map. Why?

Abstract (arXiv)

Deployed real-world machine learning applications are often subject to uncontrolled and even potentially malicious inputs. Those out-of-domain inputs can lead to unpredictable outputs and sometimes catastrophic safety issues. Prior studies on out-of-domain detection require in-domain task labels and are limited to supervised classification scenarios. Our work tackles the problem of detecting out-of-domain samples with only unsupervised in-domain data. We utilize the latent representations of pre-trained transformers and propose a simple yet effective method to transform features across all layers to construct out-of-domain detectors efficiently. Two domain-specific fine-tuning approaches are further proposed to boost detection accuracy. Our empirical evaluations of related methods on two datasets validate that our method greatly improves out-of-domain detection ability in a more general scenario.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 21 of its 33 references that are in the map. First posted on arXiv 2 Jun 2021.

Builds on

21 of 33 references.

References · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  8. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  9. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  10. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  11. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,472
  12. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,325
9 more
  1. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  2. 2018Deep Anomaly Detection with Outlier Exposurecited by 939
  3. 2019An Evaluation Dataset for Intent Classification and Out-of-Scope Predictioncited by 378
  4. 2020Pretrained Transformers Improve Out-of-Distribution Robustnesscited by 293
  5. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
  6. 2016Deep Structured Energy Based Models for Anomaly Detectioncited by 113
  7. 2020Discriminative Nearest Neighbor Few-Shot Intent Detection by Transferring Natural Language Inferencecited by 47
  8. 2021Incremental Few-shot Text Classification with Multi-round New Classes: Formulation, Dataset and Systemcited by 30
  9. 2018Deep Weighted Averaging Classifierscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 9; 2023: 9; 2024: 5; 2025: 6; 2026: 1.

31 papers.

Sort:

By area:

  1. 2022Delving into Out-of-Distribution Detection with Vision-Language Representationscited by 215
  2. 2022POEM: Out-of-Distribution Detection with Posterior Samplingcited by 108
  3. 2024On Large Language Model Continual Unlearningcited by 51
  4. 2022Predicting Out-of-Distribution Error with the Projection Normcited by 50
  5. 2023GL-MCM: Global and Local Maximum Concept Matching for Zero-Shot Out-of-Distribution Detectioncited by 36
  6. 2023A Survey on Out-of-Distribution Detection in NLPcited by 31
  7. 2023Is Fine-tuning Needed? Pre-trained Language Models Are Near Perfect for Out-of-Domain Detectioncited by 29
  8. 2021Are Pretrained Transformers Robust in Intent Classification? A Missing Ingredient in Evaluation of Out-of-Scope Intent Detectioncited by 22
  9. 2022Learning Discriminative Representations and Decision Boundaries for Open Intent Detectioncited by 15
  10. 2022Fine-grained Category Discovery under Coarse-grained supervision with Hierarchical Weighted Self-contrastive Learningcited by 13
  11. 2023Fine-Tuning Deteriorates General Textual Out-of-Distribution Detection by Distorting Task-Agnostic Featurescited by 12
  12. 2022Estimating Soft Labels for Out-of-Domain Intent Detectioncited by 11
19 more
  1. 2022Multi-Level Knowledge Distillation for Out-of-Distribution Detection in Textcited by 11
  2. 2024SeTAR: Out-of-Distribution Detection with Selective Low-Rank Approximationcited by 9
  3. 2025RODEO: Robust Outlier Detection via Exposing Adaptive Out-of-Distribution Samplescited by 7
  4. 2023Out-of-Domain Intent Detection Considering Multi-Turn Dialogue Contextscited by 6
  5. 2025Mysteries of the Deep: Role of Intermediate Representations in Out of Distribution Detectioncited by 6
  6. 2022Learning to Classify Open Intent via Soft Labeling and Manifold Mixupcited by 5
  7. 2023Masked Language Model Based Textual Adversarial Example Detectioncited by 5
  8. 2023FLatS: Principled Out-of-Distribution Detection with Feature-Based Likelihood Ratio Scorecited by 5
  9. 2025SimLabel: Consistency-Guided OOD Detection with Pretrained Vision-Language Modelscited by 5
  10. 2025Exploring Gradient-Guided Masked Language Model to Detect Textual Adversarial Attackscited by 4
  11. 2024Out-of-Distribution Detection through Soft Clustering with Non-Negative Kernel Regressioncited by 3
  12. 2024Transformer-based Joint Modelling for Automatic Essay Scoring and Off-Topic Detectioncited by 2
  13. 2024Improved Out-of-Scope Intent Classification with Dual Encoding and Threshold-based Re-Classificationcited by 2
  14. 2022Understanding, Detecting, and Separating Out-of-Distribution Samples and Adversarial Samples in Text Classificationcited by 1
  15. 2023Unsupervised Out-of-Distribution Dialect Detection with Mahalanobis Distancecited by 1
  16. 2023BED: Bi-Encoder-Based Detectors for Out-of-Distribution Detectioncited by 0
  17. 2025Rotation Control Unlearning: Quantifying and Controlling Continuous Unlearning for LLM with The Cognitive Rotation Spacecited by 0
  18. 2025DROID: Dual Representation for Out-of-Scope Intent Detectioncited by 0
  19. 2026Scaling Pretrained Representations Enables Label-Free Out-of-Distribution Detection Without Fine-Tuningcited by 0