mlsanityData to 14 Oct 2026

Enhancing Out-of-Distribution Detection in Natural Language Understanding via Implicit Layer Ensemble

arXiv 2210.11034 ↗First posted 20 Oct 2022cs.CL cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Out-of-distribution (OOD) detection aims to discern outliers from the intended data distribution, which is crucial to maintaining high reliability and a good user experience. Most recent studies in OOD detection utilize the information from a single representation that resides in the penultimate layer to determine whether the input is anomalous or not. Although such a method is straightforward, the potential of diverse information in the intermediate layers is overlooked. In this paper, we propose a novel framework based on contrastive learning that encourages intermediate features to learn layer-specialized representations and assembles them implicitly into a single representation to absorb rich information in the pre-trained language model. Extensive experiments in various intent classification and OOD datasets demonstrate that our approach is significantly more effective than other works.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 45 of its 57 references that are in the map. First posted on arXiv 20 Oct 2022.

Builds on

45 of 57 references.

References · 45

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  6. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  7. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  8. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  9. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  10. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  11. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,485
  12. 2020Energy-based Out-of-distribution Detectioncited by 1,299
33 more
  1. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  2. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  3. 2018Deep Anomaly Detection with Outlier Exposurecited by 942
  4. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 788
  5. 2018Predictive Uncertainty Estimation via Prior Networkscited by 557
  6. 2020CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instancescited by 441
  7. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 433
  8. 2017Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samplescited by 389
  9. 2019An Evaluation Dataset for Intent Classification and Out-of-Scope Predictioncited by 386
  10. 2018Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfacescited by 377
  11. 2018Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problemcited by 323
  12. 2021SSD: A Unified Framework for Self-Supervised Outlier Detectioncited by 264
  13. 2021ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfercited by 231
  14. 2019Assessing BERT's Syntactic Abilitiescited by 192
  15. 2020Contrastive Training for Improved Out-of-Distribution Detectioncited by 186
  16. 2018Reducing Network Agnostophobiacited by 165
  17. 2017DOC: Deep Open Classification of Text Documentscited by 109
  18. 2021Self-Guided Contrastive Learning for BERT Sentence Representationscited by 103
  19. 2021Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoderscited by 88
  20. 2020A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generationcited by 72
  21. 2021Contrastive Out-of-Distribution Detection for Pretrained Transformerscited by 71
  22. 2019Deep Unknown Intent Detection with Margin Losscited by 66
  23. 2019Out-of-domain Detection for Natural Language Understanding in Dialog Systemscited by 62
  24. 2021Modeling Discriminative Representations for Out-of-Domain Detection with Supervised Contrastive Learningcited by 42
  25. 2021Out-of-Scope Intent Detection with Self-Supervision and Discriminative Trainingcited by 42
  26. 2020MASKER: Masked Keyword Regularization for Reliable Text Classificationcited by 25
  27. 2021DATE: Detecting Anomalies in Text via Self-Supervision of Transformerscited by 25
  28. 2020SLM: Learning a Discourse Language Representation with Sentence Unshufflingcited by 24
  29. 2021Masked Contrastive Learning for Anomaly Detectioncited by 23
  30. 2021Are Pretrained Transformers Robust in Intent Classification? A Missing Ingredient in Evaluation of Out-of-Scope Intent Detectioncited by 22
  31. 2021Enhancing the Generalization for Intent Classification and Out-of-Domain Detection in SLUcited by 22
  32. 2021GOLD: Improving Out-of-Scope Detection in Dialogues using Data Augmentationcited by 20
  33. 2020Cross-Domain Sentiment Classification with Contrastive Learning and Mutual Information Maximizationcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 3; 2025: 0; 2026: 1.

9 papers.

  1. 2023A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validationcited by 93
  2. 2023A Survey on Out-of-Distribution Detection in NLPcited by 31
  3. 2024Out-of-Distribution Detection via Deep Multi-Comprehension Ensemblecited by 8
  4. 2023Revealing User Familiarity Bias in Task-Oriented Dialogue via Interactive Evaluationcited by 4
  5. 2023Probing Out-of-Distribution Robustness of Language Models with Parameter-Efficient Transfer Learningcited by 2
  6. 2023Universal Domain Adaptation for Robust Handling of Distributional Shifts in NLPcited by 2
  7. 2024Inner-Probe: Discovering Copyright-related Data Generation in LLM Architecturecited by 1
  8. 2026ReIn: Conversational Error Recovery with Reasoning Inceptioncited by 1
  9. 2024VI-OOD: A Unified Representation Learning Framework for Textual Out-of-distribution Detectioncited by 0