mlsanityData to 14 Oct 2026

Measuring the Robustness of NLP Models to Domain Shifts

arXiv 2306.00168 ↗First posted 31 May 2023cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

Existing research on Domain Robustness (DR) suffers from disparate setups, limited task variety, and scarce research on recent capabilities such as in-context learning. Furthermore, the common practice of measuring DR might not be fully accurate. Current research focuses on challenge sets and relies solely on the Source Drop (SD): Using the source in-domain performance as a reference point for degradation. However, we argue that the Target Drop (TD), which measures degradation from the target in-domain performance, should be used as a complementary point of view. To address these issues, we first curated a DR benchmark comprised of 7 diverse NLP tasks, which enabled us to measure both the SD and the TD. We then conducted a comprehensive large-scale DR study involving over 14,000 domain shifts across 21 fine-tuned models and few-shot LLMs. We found that both model types suffer from drops upon domain shifts. While fine-tuned models excel in-domain, few-shot LLMs often surpass them cross-domain, showing better robustness. In addition, we found that a large SD can often be explained by shifting to a harder domain rather than by a genuine DR challenge, and this highlights the importance of TD as a complementary metric. We hope our study will shed light on the current DR state of NLP models and promote improved evaluation practices toward more robust models.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 58 of its 104 references that are in the map. First posted on arXiv 31 May 2023.

Builds on

58 of 104 references.

References · 58

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  10. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  11. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  12. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
38 more
  1. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  2. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  3. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  4. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,375
  5. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  6. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  7. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
  8. 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 766
  9. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 671
  10. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 439
  11. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 365
  12. 2023A Comprehensive Survey on Test-Time Adaptation under Distribution Shiftscited by 359
  13. 2017Synthetic and Natural Noise Both Break Neural Machine Translationcited by 306
  14. 2020Pretrained Transformers Improve Out-of-Distribution Robustnesscited by 294
  15. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  16. 2022Data Contamination: From Memorization to Exploitationcited by 236
  17. 2022What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?cited by 150
  18. 2021On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptationcited by 147
  19. 2020Neural Unsupervised Domain Adaptation in NLP---A Surveycited by 144
  20. 2023Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluationcited by 136
  21. 2020An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Modelscited by 134
  22. 2020A survey on domain adaptation theory: learning bounds and theoretical guaranteescited by 109
  23. 2021Robustness Gym: Unifying the NLP Evaluation Landscapecited by 108
  24. 2021Measure and Improve Robustness in NLP Models: A Surveycited by 107
  25. 2020The Effect of Natural Distribution Shift on Question Answering Modelscited by 104
  26. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 98
  27. 2022Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Againcited by 83
  28. 2021PADA: Example-based Prompt Learning for on-the-fly Adaptation to Unseen Domainscited by 61
  29. 2021AdaptSum: Towards Low-Resource Domain Adaptation for Abstractive Summarizationcited by 51
  30. 2022DoCoGen: Domain Counterfactual Generation for Low Resource Domain Adaptationcited by 37
  31. 2020PERL: Pivot-based Domain Adaptation for Pre-trained Deep Contextualized Embedding Modelscited by 33
  32. 2022Standing on the Shoulders of Giant Frozen Language Modelscited by 33
  33. 2023A Survey on Out-of-Distribution Detection in NLPcited by 31
  34. 2022M2D2: A Massively Multi-domain Language Modeling Datasetcited by 26
  35. 2023CombLM: Adapting Black-Box Language Models through Small Fine-Tuned Modelscited by 26
  36. 2019Deep Contextualized Self-training for Low Resource Dependency Parsingcited by 24
  37. 2020Domain Divergences: a Survey and Empirical Analysiscited by 24
  38. 2020Towards Improving Selective Prediction Ability of NLP Systemscited by 20

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 4; 2026: 5.

11 papers.

  1. 2024On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMscited by 25
  2. 2024Are LLMs Better than Reported? Detecting Label Errors and Mitigating Their Effect on Model Performancecited by 20
  3. 2025AdaptiVocab: Enhancing LLM Efficiency in Focused Domains through Lightweight Vocabulary Adaptationcited by 15
  4. 2025Adapting Large Language Models for Multi-Domain Retrieval-Augmented-Generationcited by 3
  5. 2025Multi-Domain Explainability of Preferencescited by 2
  6. 2026From Token to Token Pair: Efficient Prompt Compression for Large Language Models in Clinical Predictioncited by 2
  7. 2025Relation Extraction or Pattern Matching? Unravelling the Generalisation Limits of Language Models for Biographical REcited by 1
  8. 2026Evaluating Black-Box Vulnerabilities with Wasserstein-Constrained Data Perturbationscited by 0
  9. 2026Robust Biomedical Publication Type and Study Design Classification with Knowledge-Guided Perturbationscited by 0
  10. 2026What Transfers Under Source Shift? Definitions, Examples, and Fine-Tuning for Climate Disclosure Classificationcited by 0
  11. 2026When to Call an LLM: A Confidence-Gated Hybrid for Cost-Effective Emotion Recognition in Conversational AIcited by 0