mlsanityData to 7 Oct 2026

State-of-the-art generalisation research in NLP: A taxonomy and review

arXiv 2210.03050 ↗First posted 6 Oct 2022cs.CL cs.AIcited by 97

Not on the map. Why?

Abstract (arXiv)

The ability to generalise well is one of the primary desiderata of natural language processing (NLP). Yet, what 'good generalisation' entails and how it should be evaluated is not well understood, nor are there any evaluation standards for generalisation. In this paper, we lay the groundwork to address both of these issues. We present a taxonomy for characterising and understanding generalisation research in NLP. Our taxonomy is based on an extensive literature review of generalisation research, and contains five axes along which studies can differ: their main motivation, the type of generalisation they investigate, the type of data shift they consider, the source of this data shift, and the locus of the shift within the modelling pipeline. We use our taxonomy to classify over 400 papers that test generalisation, for a total of more than 600 individual experiments. Considering the results of this review, we present an in-depth analysis that maps out the current state of generalisation research in NLP, and we make recommendations for which areas might deserve attention in the future. Along with this paper, we release a webpage where the results of our review can be dynamically explored, and which we intend to update as new NLP generalisation studies are published. With this work, we aim to take steps towards making state-of-the-art generalisation testing the new status quo in NLP.

Key facts. Cited by 97 papers in the map (data through 7 Oct 2026). Builds on 469 of its 697 references that are in the map. First posted on arXiv 6 Oct 2022.

Builds on

469 of 697 references.

References · 469

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2016Pointer Sentinel Mixture Modelscited by 2,992
  10. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  11. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  12. 2018Deep contextualized word representationscited by 2,285
38 more
  1. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  2. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  3. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  4. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  5. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  6. 2018Datasheets for Datasetscited by 1,371
  7. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  8. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  9. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  10. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  11. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  12. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  13. 2018Model Cards for Model Reportingcited by 1,068
  14. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  15. 2019How Can We Know What Language Models Know?cited by 953
  16. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  17. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  18. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  19. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  20. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  21. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  22. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  23. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  24. 2019How multilingual is Multilingual BERT?cited by 642
  25. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  26. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  27. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  28. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  29. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  30. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 553
  31. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  32. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  33. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  34. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  35. 2021Towards Out-Of-Distribution Generalization: A Surveycited by 431
  36. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 428
  37. 2017Zero-Shot Relation Extraction via Reading Comprehensioncited by 425
  38. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 423

50 most cited of 469 All 469 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 28; 2024: 25; 2025: 28; 2026: 13.

97 papers.

Sort:

By area:

  1. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  2. 2023Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solvecited by 175
  3. 2022Diverse Demonstrations Improve In-context Compositional Generalizationcited by 136
  4. 2023Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluationcited by 134
  5. 2024THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulationcited by 128
  6. 2023Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Modelscited by 117
  7. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  8. 2023Uncertainty in Natural Language Generation: From Theory to Applicationscited by 61
  9. 2024Detection and Measurement of Syntactic Templates in Generated Textcited by 44
  10. 2023In-Context Learning for Text Classification with Many Labelscited by 35
  11. 2024Filtered Corpus Training (FiCT) Shows that Language Models can Generalize from Indirect Evidencecited by 35
  12. 2023Mind the instructions: a holistic evaluation of consistency and interactions in prompt-based learningcited by 31
38 more
  1. 2024Multilingual Pretraining and Instruction Tuning Improve Cross-Lingual Knowledge Alignment, But Only Shallowlycited by 27
  2. 2022Language model acceptability judgements are not always robust to contextcited by 22
  3. 2024A Philosophical Introduction to Language Models -- Part I: Continuity With Classic Debatescited by 22
  4. 2025MultiLoKo: a multilingual local knowledge benchmark for LLMs spanning 31 languagescited by 22
  5. 2023Interpretable Word Sense Representations via Definition Generation: The Case of Semantic Change Analysiscited by 19
  6. 2024Data Contamination Report from the 2024 CONDA Shared Taskcited by 19
  7. 2023Viewing Knowledge Transfer in Multilingual Machine Translation Through a Representational Lenscited by 15
  8. 2023Separating form and meaning: Using self-consistency to quantify task understanding across multiple sensescited by 15
  9. 2023Walking a Tightrope -- Evaluating Large Language Models in High-Risk Domainscited by 15
  10. 2024A Survey on Compositional Learning of AI Models: Theoretical and Experimental Practicescited by 15
  11. 2024Tokenization and Morphology in Multilingual Language Models: A Comparative Analysis of mT5 and ByT5cited by 14
  12. 2025Numerical models outperform AI weather forecasts of record-breaking extremescited by 14
  13. 2023What makes a language easy to deep-learn? Deep neural networks and humans similarly benefit from compositional structurecited by 13
  14. 2023The Validity of Evaluation Results: Assessing Concurrence Across Compositionality Benchmarkscited by 12
  15. 2023Measuring the Robustness of NLP Models to Domain Shiftscited by 11
  16. 2025How Causal Abstraction Underpins Computational Explanationcited by 10
  17. 2024Amuro and Char: Analyzing the Relationship between Pre-Training and Fine-Tuning of Large Language Modelscited by 9
  18. 2025Gradient-Guided Annealing for Domain Generalizationcited by 9
  19. 2024A Systematic Review of Generalization Research in Medical Image Classificationcited by 8
  20. 2024Evaluating Subword Tokenization: Alien Subword Composition and OOV Generalization Challengecited by 8
  21. 2024Investigating the Role of Instruction Variety and Task Difficulty in Robotic Manipulation Taskscited by 8
  22. 2024From Form(s) to Meaning: Probing the Semantic Depths of Language Models Using Multisense Consistencycited by 7
  23. 2024From Frege to chatGPT: Compositionality in language, cognition, and deep neural networkscited by 7
  24. 2024Creativity in AI: Progresses and Challengescited by 7
  25. 2025Measuring Combinatorial Creativity in Large Language Models and Humanscited by 7
  26. 2025MoVa: Towards Generalizable Classification of Human Morals and Valuescited by 7
  27. 2022Assessing Out-of-Domain Language Model Performance from Few Examplescited by 6
  28. 2023Investigating Multi-source Active Learning for Natural Language Inferencecited by 6
  29. 2023GenCodeSearchNet: A Benchmark Test Suite for Evaluating Generalization in Programming Language Understandingcited by 6
  30. 2023The ICL Consistency Testcited by 6
  31. 2025Diagnosing Moral Reasoning Acquisition in Language Models: Pragmatics and Generalizationcited by 6
  32. 2025Domain Regeneration: How well do LLMs match syntactic properties of text domains?cited by 6
  33. 2026Emergent and Subliminal Misalignment Through the Lens of Data-Mediated Transfercited by 6
  34. 2023Classical Out-of-Distribution Detection Methods Benchmark in Text Classification Taskscited by 5
  35. 2023Whispers of Doubt Amidst Echoes of Triumph in NLP Robustnesscited by 5
  36. 2024Frequency matters: Modeling irregular morphological patterns in Spanish with Transformerscited by 5
  37. 2024Quantifying artificial intelligence through algorithmic generalizationcited by 5
  38. 2025Are Knowledge and Reference in Multilingual Language Models Cross-Lingually Consistent?cited by 5

50 most cited of 97 All 97 (JSON)