Probing Out-of-Distribution Robustness of Language Models with Parameter-Efficient Transfer Learning
Not on the map. Why?
Abstract (arXiv)
As the size of the pre-trained language model (PLM) continues to increase, numerous parameter-efficient transfer learning methods have been proposed recently to compensate for the tremendous cost of fine-tuning. Despite the impressive results achieved by large pre-trained language models (PLMs) and various parameter-efficient transfer learning (PETL) methods on sundry benchmarks, it remains unclear if they can handle inputs that have been distributionally shifted effectively. In this study, we systematically explore how the ability to detect out-of-distribution (OOD) changes as the size of the PLM grows or the transfer methods are altered. Specifically, we evaluated various PETL techniques, including fine-tuning, Adapter, LoRA, and prefix-tuning, on three different intention classification tasks, each utilizing various language models with different scales.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 44 of its 55 references that are in the map. First posted on arXiv 27 Jan 2023.
Builds on
44 of 55 references.
References · 44
- 2017Attention Is All You Needcited by 60,361
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2016Layer Normalizationcited by 5,669
- 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
- 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
- 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
- 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
- 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
32 more
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
- 2022Training Compute-Optimal Large Language Modelscited by 3,019
- 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
- 2020Shortcut Learning in Deep Neural Networkscited by 1,969
- 2019Language Models as Knowledge Bases?cited by 1,936
- 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
- 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,485
- 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
- 2020Energy-based Out-of-distribution Detectioncited by 1,299
- 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
- 2019How Can We Know What Language Models Know?cited by 959
- 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 938
- 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 788
- 2021GPT Understands, Toocited by 716
- 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
- 2020CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instancescited by 441
- 2020Efficient Intent Detection with Dual Sentence Encoderscited by 433
- 2019An Evaluation Dataset for Intent Classification and Out-of-Scope Predictioncited by 386
- 2020AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Promptscited by 249
- 2022Synchromesh: Reliable code generation from pre-trained language modelscited by 156
- 2017DOC: Deep Open Classification of Text Documentscited by 109
- 2021Self-Guided Contrastive Learning for BERT Sentence Representationscited by 103
- 2021Contrastive Out-of-Distribution Detection for Pretrained Transformerscited by 71
- 2021Revisiting Mahalanobis Distance for Transformer-Based Out-of-Domain Detectioncited by 70
- 2019Deep Unknown Intent Detection with Margin Losscited by 66
- 2020SBERT-WK: A Sentence Embedding Method by Dissecting BERT-based Word Modelscited by 56
- 2021Modeling Discriminative Representations for Out-of-Domain Detection with Supervised Contrastive Learningcited by 42
- 2020MASKER: Masked Keyword Regularization for Reliable Text Classificationcited by 25
- 2021Are Pretrained Transformers Robust in Intent Classification? A Missing Ingredient in Evaluation of Out-of-Scope Intent Detectioncited by 22
- 2021Enhancing the Generalization for Intent Classification and Out-of-Domain Detection in SLUcited by 22
- 2020Cross-Domain Sentiment Classification with Contrastive Learning and Mutual Information Maximizationcited by 9
- 2022Enhancing Out-of-Distribution Detection in Natural Language Understanding via Implicit Layer Ensemblecited by 9
Built on by
Citing papers per year: 2023: 1; 2024: 1.
2 papers.