mlsanityData to 7 Oct 2026

Dive into the Chasm: Probing the Gap between In- and Cross-Topic Generalization

arXiv 2402.01375 ↗First posted 2 Feb 2024cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

Pre-trained language models (LMs) perform well in In-Topic setups, where training and testing data come from the same topics. However, they face challenges in Cross-Topic scenarios where testing data is derived from distinct topics -- such as Gun Control. This study analyzes various LMs with three probing-based experiments to shed light on the reasons behind the In- vs. Cross-Topic generalization gap. Thereby, we demonstrate, for the first time, that generalization gaps and the robustness of the embedding space vary significantly across LMs. Additionally, we assess larger LMs and underscore the relevance of our analysis for recent models. Overall, diverse pre-training objectives, architectural regularization, or data deduplication contribute to more robust LMs and diminish generalization gaps. Our research contributes to a deeper understanding and comparison of language models across different generalization scenarios.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 38 of its 58 references that are in the map. First posted on arXiv 2 Feb 2024.

Builds on

38 of 58 references.

References · 38

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  8. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  9. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  10. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  11. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  12. 2020In Search of Lost Domain Generalizationcited by 1,042
26 more
  1. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  2. 2019Designing and Interpreting Probes with Control Taskscited by 608
  3. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  4. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
  5. 2021Towards Out-Of-Distribution Generalization: A Surveycited by 431
  6. 2020Information-Theoretic Probing with Minimum Description Lengthcited by 230
  7. 2020Information-Theoretic Probing for Linguistic Structurecited by 204
  8. 2017What do Neural Machine Translation Models Learn about Morphology?cited by 165
  9. 2019oLMpics -- On what Language Model Pre-training Capturescited by 161
  10. 2019Do NLP Models Know Numbers? Probing Numeracy in Embeddingscited by 157
  11. 2018Dissecting Contextual Word Embeddings: Architecture and Representationcited by 146
  12. 2020Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?cited by 144
  13. 2020What Happens To BERT Embeddings During Fine-tuning?cited by 113
  14. 2020Zero-Shot Stance Detection: A Dataset and Model using Generalized Topic Representationscited by 97
  15. 2022GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-distribution Generalization Perspectivecited by 72
  16. 2018Cross-topic Argument Mining from Heterogeneous Sources Using Attention-based Neural Networkscited by 58
  17. 2022Probing for the Usage of Grammatical Numbercited by 57
  18. 2020Will-They-Won't-They: A Very Large Dataset for Stance Detection on Twittercited by 53
  19. 2021Temporal Adaptation of BERT and Performance on Downstream Document Classification: Insights from Social Mediacited by 51
  20. 2021A Closer Look at How Fine-tuning Changes BERTcited by 42
  21. 2020On the Interplay Between Fine-tuning and Sentence-level Probing for Linguistic Knowledge in Pre-trained Transformerscited by 39
  22. 2021Discourse Probing of Pretrained Language Modelscited by 38
  23. 2021Cross-Domain Label-Adaptive Stance Detectioncited by 30
  24. 2022Metaphors in Pre-Trained Language Models: Probing and Generalization Across Datasets and Languagescited by 30
  25. 2021Is Stance Detection Topic-Independent and Cross-topic Generalizable? -- A Reproduction Studycited by 9
  26. 2022OOD-Probe: A Neural Interpretation of Out-of-Domain Generalizationcited by 5

Built on by

Cited by year

Citing papers per year: 2024: 4; 2025: 2.

6 papers.

  1. 2024Holmes: A Benchmark to Assess the Linguistic Competence of Language Modelscited by 26
  2. 2024What Variables Affect Out-of-Distribution Generalization in Pretrained Models?cited by 13
  3. 2025Controlling Neural Collapse Enhances Out-of-Distribution Detection and Transfer Learningcited by 11
  4. 2025Aligned Probing: Relating Toxic Behavior and Model Internalscited by 5
  5. 2024Investigating the Robustness of Modelling Decisions for Few-Shot Cross-Topic Stance Detection: A Preregistered Studycited by 1
  6. 2024Is the Digital Forensics and Incident Response Pipeline Ready for Text-Based Threats in LLM Era?cited by 1