NoiER: An Approach for Training more Reliable Fine-TunedDownstream Task Models
Not on the map. Why?
Abstract (arXiv)
The recent development in pretrained language models trained in a self-supervised fashion, such as BERT, is driving rapid progress in the field of NLP. However, their brilliant performance is based on leveraging syntactic artifacts of the training data rather than fully understanding the intrinsic meaning of language. The excessive exploitation of spurious artifacts causes a problematic issue: The distribution collapse problem, which is the phenomenon that the model fine-tuned on downstream tasks is unable to distinguish out-of-distribution (OOD) sentences while producing a high confidence score. In this paper, we argue that distribution collapse is a prevalent issue in pretrained language models and propose noise entropy regularisation (NoiER) as an efficient learning paradigm that solves the problem without auxiliary models and additional~data. The proposed approach improved traditional OOD detection evaluation metrics by 55% on average compared to the original fine-tuned models.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 28 of its 43 references that are in the map. First posted on arXiv 29 Aug 2021.
Builds on
28 of 43 references.
References · 28
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2014Explaining and Harnessing Adversarial Examplescited by 7,364
- 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
- 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
- 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
- 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
- 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
- 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
- 2018Deep contextualized word representationscited by 2,290
- 2016Concrete Problems in AI Safetycited by 1,642
- 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,328
- 2018Deep Anomaly Detection with Outlier Exposurecited by 942
16 more
- 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
- 2017Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samplescited by 389
- 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
- 2017A Structured Self-attentive Sentence Embeddingcited by 318
- 2017Unsupervised Machine Translation Using Monolingual Corpora Onlycited by 273
- 2019ERNIE 2.0: A Continual Pre-training Framework for Language Understandingcited by 231
- 2019Probing Neural Network Comprehension of Natural Language Argumentscited by 214
- 2019Out-of-domain Detection for Natural Language Understanding in Dialog Systemscited by 62
- 2017The Argument Reasoning Comprehension Task: Identification and Reconstruction of Implicit Warrantscited by 56
- 2020SqueezeBERT: What can computer vision teach NLP about efficient neural networks?cited by 50
- 2020Generalized Entropy Regularization or: There's Nothing Special about Label Smoothingcited by 41
- 2018Open-world Learning and Application to Product Classificationcited by 39
- 2018Neural Sentence Embedding using Only In-domain Sentences for Out-of-domain Sentence Detection in Dialog Systemscited by 18
- 2019A Simple but Effective Method to Incorporate Multi-turn Context with BERT for Conversational Machine Comprehensioncited by 18
- 2018Joint Learning of Domain Classification and Out-of-Domain Detection with Dynamic Class Weighting for Satisficing False Acceptance Ratescited by 17
- 2018Word Embedding Perturbation for Sentence Classificationcited by 11
Built on by
Citing papers per year: 2021: 1.
1 paper.