mlsanityData to 7 Oct 2026

GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-distribution Generalization Perspective

arXiv 2211.08073 ↗First posted 15 Nov 2022cs.CL cs.AI cs.LGcited by 72

Not on the map. Why?

Abstract (arXiv)

Pre-trained language models (PLMs) are known to improve the generalization performance of natural language understanding models by leveraging large amounts of data during the pre-training phase. However, the out-of-distribution (OOD) generalization problem remains a challenge in many NLP tasks, limiting the real-world deployment of these methods. This paper presents the first attempt at creating a unified benchmark named GLUE-X for evaluating OOD robustness in NLP models, highlighting the importance of OOD robustness and providing insights on how to measure the robustness of a model and how to improve it. The benchmark includes 13 publicly available datasets for OOD testing, and evaluations are conducted on 8 classic NLP tasks over 21 popularly used PLMs, including GPT-3 and GPT-3.5. Our findings confirm the need for improved OOD accuracy in NLP tasks, as significant performance degradation was observed in all settings compared to in-distribution (ID) accuracy.

Key facts. Cited by 72 papers in the map (data through 7 Oct 2026). Builds on 78 of its 102 references that are in the map. First posted on arXiv 15 Nov 2022.

Builds on

78 of 102 references.

References · 78

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  10. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  11. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  12. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
38 more
  1. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  2. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  3. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  4. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  5. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  6. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  7. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  8. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  9. 2021Generalizing to Unseen Domains: A Survey on Domain Generalizationcited by 762
  10. 2020Out-of-Distribution Generalization via Risk Extrapolation (REx)cited by 737
  11. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  12. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  13. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  14. 2013Domain Generalization via Invariant Feature Representationcited by 622
  15. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 590
  16. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  17. 2019ERASER: A Benchmark to Evaluate Rationalized NLP Modelscited by 523
  18. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 504
  19. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  20. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  21. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 428
  22. 2020Self-Challenging Improves Cross-Domain Generalizationcited by 426
  23. 2019Domain Generalization via Model-Agnostic Learning of Semantic Featurescited by 405
  24. 2016NewsQA: A Machine Comprehension Datasetcited by 330
  25. 2020An Investigation of Why Overparameterization Exacerbates Spurious Correlationscited by 310
  26. 2020Pretrained Transformers Improve Out-of-Distribution Robustnesscited by 293
  27. 2016Rationalizing Neural Predictionscited by 281
  28. 2020CLUE: A Chinese Language Understanding Evaluation Benchmarkcited by 277
  29. 2022OpenOOD: Benchmarking Generalized Out-of-Distribution Detectioncited by 267
  30. 2021Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Modelscited by 227
  31. 2021Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Modelscited by 219
  32. 2019Probing Neural Network Comprehension of Natural Language Argumentscited by 212
  33. 2020On the Theory of Transfer Learning: The Importance of Task Diversitycited by 186
  34. 2022GOOD: A Graph Out-of-Distribution Benchmarkcited by 140
  35. 2020An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Modelscited by 133
  36. 2017Bilateral Multi-Perspective Matching for Natural Language Sentencescited by 120
  37. 2021Systematic Inequalities in Language Technology Performance across the World's Languagescited by 118
  38. 2020Understanding and Improving Information Transfer in Multi-Task Learningcited by 113

50 most cited of 78 All 78 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 26; 2024: 27; 2025: 13; 2026: 5.

72 papers.

Sort:

By area:

  1. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  2. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  3. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 218
  4. 2023On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspectivecited by 208
  5. 2023DyVal: Dynamic Evaluation of Large Language Models for Reasoning Taskscited by 100
  6. 2023Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluationscited by 96
  7. 2024A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendationscited by 84
  8. 2024T2VSafetyBench: Evaluating the Safety of Text-to-Video Generative Modelscited by 66
  9. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 63
  10. 2024Dynamic Evaluation of Large Language Models by Meta Probing Agentscited by 54
  11. 2024KIEval: A Knowledge-grounded Interactive Evaluation Framework for Large Language Modelscited by 36
  12. 2024Benchmarks as Microscopes: A Call for Model Metrologycited by 33
38 more
  1. 2024Can Large Language Models Understand Symbolic Graphics Programs?cited by 32
  2. 2024Examining the robustness of LLM evaluation to the distributional assumptions of benchmarkscited by 31
  3. 2023Understanding and Mitigating the Label Noise in Pre-training on Downstream Taskscited by 27
  4. 2025RewardAnything: Generalizable Principle-Following Reward Modelscited by 27
  5. 2024Holmes: A Benchmark to Assess the Linguistic Competence of Language Modelscited by 26
  6. 2024Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generationcited by 21
  7. 2023Rethinking the Evaluation Protocol of Domain Generalizationcited by 20
  8. 2023Supervised Knowledge Makes Large Language Models Better In-context Learnerscited by 20
  9. 2024A Survey on Evaluation of Out-of-Distribution Generalizationcited by 20
  10. 2025An Empirical Analysis of Uncertainty in Large Language Model Evaluationscited by 20
  11. 2023Robust Prompt Optimization for Large Language Models Against Distribution Shiftscited by 19
  12. 2024A Closer Look at Benchmarking Self-Supervised Pre-training with Image Classificationcited by 19
  13. 2023Unlocking Emergent Modularity in Large Language Modelscited by 17
  14. 2024Evaluating the Generalization Ability of Quantized LLMs: Benchmark, Analysis, and Toolboxcited by 11
  15. 2022Prototypical Fine-tuning: Towards Robust Performance Under Varying Data Sizescited by 10
  16. 2024SelfPrompt: Autonomously Evaluating LLM Robustness via Domain-Constrained Knowledge Guidelines and Refined Adversarial Promptscited by 10
  17. 2023Generalization Analogies: A Testbed for Generalizing AI Oversight to Hard-To-Measure Domainscited by 8
  18. 2024Impact of Noisy Supervision in Foundation Model Learningcited by 8
  19. 2023On the Usage of Continual Learning for Out-of-Distribution Generalization in Pre-trained Language Models of Codecited by 7
  20. 2024FreeEval: A Modular Framework for Trustworthy and Efficient Evaluation of Large Language Modelscited by 7
  21. 2024BA-LoRA: Bias-Alleviating Low-Rank Adaptation to Mitigate Catastrophic Inheritance in Large Language Modelscited by 7
  22. 2025CLINIC: Evaluating Multilingual Trustworthiness in Language Models for Healthcarecited by 7
  23. 2024Dive into the Chasm: Probing the Gap between In- and Cross-Topic Generalizationcited by 6
  24. 2025Reasoning is All You Need for Video Generalization: A Counterfactual Benchmark with Sub-question Evaluationcited by 6
  25. 2023Classical Out-of-Distribution Detection Methods Benchmark in Text Classification Taskscited by 5
  26. 2023Whispers of Doubt Amidst Echoes of Triumph in NLP Robustnesscited by 5
  27. 2024A Rationale-centric Counterfactual Data Augmentation Method for Cross-Document Event Coreference Resolutioncited by 5
  28. 2025None of the Others: a General Technique to Distinguish Reasoning from Memorization in Multiple-Choice LLM Evaluation Benchmarkscited by 5
  29. 2025SAEMark: Steering Personalized Multilingual LLM Watermarks with Sparse Autoencoderscited by 5
  30. 2023Competence-Based Analysis of Language Modelscited by 4
  31. 2023How to Handle Different Types of Out-of-Distribution Scenarios in Computational Argumentation? A Comprehensive and Fine-Grained Field Studycited by 4
  32. 2023GeSS: Benchmarking Geometric Deep Learning under Scientific Applications with Distribution Shiftscited by 4
  33. 2025A Theoretical Framework for OOD Robustness in Transformers using Gevrey Classescited by 4
  34. 2025Too Easily Fooled? Prompt Injection Breaks LLMs on Frustratingly Simple Multiple-Choice Questionscited by 4
  35. 2023Consistency Regularization for Domain Generalization with Logit Attribution Matchingcited by 3
  36. 2023From Adversarial Arms Race to Model-centric Evaluation: Motivating a Unified Automatic Robustness Evaluation Frameworkcited by 3
  37. 2024An Empirical Study on Large Language Models in Accuracy and Robustness under Chinese Industrial Scenarioscited by 3
  38. 2024KGPA: Robustness Evaluation for Large Language Models via Cross-Domain Knowledge Graphscited by 3

50 most cited of 72 All 72 (JSON)