mlsanityData to 7 Oct 2026

Beyond Mahalanobis-Based Scores for Textual OOD Detection

arXiv 2211.13527 ↗First posted 24 Nov 2022cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

Deep learning methods have boosted the adoption of NLP systems in real-life applications. However, they turn out to be vulnerable to distribution shifts over time which may cause severe dysfunctions in production systems, urging practitioners to develop tools to detect out-of-distribution (OOD) samples through the lens of the neural network. In this paper, we introduce TRUSTED, a new OOD detector for classifiers based on Transformer architectures that meets operational requirements: it is unsupervised and fast to compute. The efficiency of TRUSTED relies on the fruitful idea that all hidden layers carry relevant information to detect OOD examples. Based on this, for a given input, TRUSTED consists in (i) aggregating this information and (ii) computing a similarity score by exploiting the training distribution, leveraging the powerful concept of data depth. Our extensive numerical experiments involve 51k model configurations, including various checkpoints, seeds, and datasets, and demonstrate that TRUSTED achieves state-of-the-art performances. In particular, it improves previous AUROC over 3 points.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 57 of its 103 references that are in the map. First posted on arXiv 24 Nov 2022.

Builds on

57 of 103 references.

References · 57

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  6. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  7. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  8. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,911
  9. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,472
  10. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,325
  11. 2020Energy-based Out-of-distribution Detectioncited by 1,296
  12. 2018Deep Anomaly Detection with Outlier Exposurecited by 939
38 more
  1. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 488
  2. 2019Likelihood Ratios for Out-of-Distribution Detectioncited by 451
  3. 2020Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Datacited by 408
  4. 2020Pretrained Transformers Improve Out-of-Distribution Robustnesscited by 293
  5. 2021Unsolved Problems in ML Safetycited by 291
  6. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
  7. 2020A Survey on the Explainability of Supervised Machine Learningcited by 225
  8. 2020Fairness in Machine Learningcited by 194
  9. 2020Contrastive Training for Improved Out-of-Distribution Detectioncited by 186
  10. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 176
  11. 2021The MultiBERTs: BERT Reproductions for Robustness Analysiscited by 89
  12. 2021Types of Out-of-Distribution Texts and How to Detect Themcited by 80
  13. 2022Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generationcited by 73
  14. 2021Contrastive Out-of-Distribution Detection for Pretrained Transformerscited by 70
  15. 2021Revisiting Mahalanobis Distance for Transformer-Based Out-of-Domain Detectioncited by 67
  16. 2019Affect-Driven Dialog Generationcited by 50
  17. 2019Detecting semantic anomaliescited by 50
  18. 2015Massively Multitask Networks for Drug Discoverycited by 49
  19. 2018Convergence and Dynamical Behavior of the ADAM Algorithm for Non-Convex Stochastic Optimizationcited by 47
  20. 2021Automatic Text Evaluation through the Lens of Wasserstein Barycenterscited by 38
  21. 2021A Novel Estimator of Mutual Information for Learning to Disentangle Textual Representationscited by 37
  22. 2020Hierarchical Pre-training for Sequence Labelling in Spoken Dialogcited by 36
  23. 2021Are Larger Pretrained Language Models Uniformly Better? Comparing Performance at the Instance Levelcited by 34
  24. 2019Likelihood Ratios and Generative Classifiers for Unsupervised Out-of-Domain Detection In Task Oriented Dialogcited by 33
  25. 2021InfoLM: A New Metric to Evaluate Summarization & Data2Text Generationcited by 33
  26. 2017Anomaly Detection by Robust Statisticscited by 22
  27. 2018Concatenated Power Mean Word Embeddings as Universal Cross-Lingual Sentence Representationscited by 20
  28. 2020Heavy-tailed Representations, Text Polarity Classification & Data Augmentationcited by 20
  29. 2022A Differential Entropy Estimator for Training Neural Networkscited by 20
  30. 2022Igeood: An Information Geometry Approach to Out-of-Distribution Detectioncited by 20
  31. 2018Security and Privacy Issues in Deep Learningcited by 19
  32. 2019Functional Isolation Forestcited by 19
  33. 2021A Pseudo-Metric between Probability Distributions based on Depth-Trimmed Regionscited by 17
  34. 2020The importance of fillers for text representations of speech transcriptscited by 15
  35. 2021Improving Multimodal fusion via Mutual Dependency Maximisationcited by 15
  36. 2019From the Token to the Review: A Hierarchical Multimodal approach to Opinion Miningcited by 14
  37. 2022Learning Disentangled Textual Representations via Statistical Measures of Similaritycited by 14
  38. 2021Beam Search with Bidirectional Strategies for Neural Response Generationcited by 12

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 1; 2024: 1.

3 papers.

  1. 2022Rainproof: An Umbrella To Shield Text Generators From Out-Of-Distribution Datacited by 21
  2. 2023Detecting out-of-distribution text using topological features of transformer-based language modelscited by 2
  3. 2024One Stone, Four Birds: A Comprehensive Solution for QA System Using Supervised Contrastive Learningcited by 0