mlsanityData to 14 Oct 2026

Lexical Generalization Improves with Larger Models and Longer Training

arXiv 2210.12673 ↗First posted 23 Oct 2022cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

While fine-tuned language models perform well on many tasks, they were also shown to rely on superficial surface features such as lexical overlap. Excessive utilization of such heuristics can lead to failure on challenging inputs. We analyze the use of lexical overlap heuristics in natural language inference, paraphrase detection, and reading comprehension (using a novel contrastive dataset), and find that larger models are much less susceptible to adopting lexical overlap heuristics. We also find that longer training leads models to abandon lexical overlap heuristics. Finally, we provide evidence that the disparity between models size has its source in the pre-trained model

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 42 of its 64 references that are in the map. First posted on arXiv 23 Oct 2022.

Builds on

42 of 64 references.

References · 42

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  4. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  5. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  6. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  7. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  8. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  9. 2018Relational inductive biases, deep learning, and graph networkscited by 1,360
  10. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  11. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
  12. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 571
30 more
  1. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 460
  2. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 402
  3. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 384
  4. 2017Synthetic and Natural Noise Both Break Neural Machine Translationcited by 306
  5. 2018Hypothesis Only Baselines in Natural Language Inferencecited by 295
  6. 2018Adversarial Example Generation with Syntactically Controlled Paraphrase Networkscited by 286
  7. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 220
  8. 2018Stress Test Evaluation for Natural Language Inferencecited by 213
  9. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 202
  10. 2019BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Modelcited by 185
  11. 2020An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Modelscited by 134
  12. 2019Unlearn Dataset Bias in Natural Language Inference by Fitting the Residualcited by 119
  13. 2020Towards Debiasing NLU Models from Unknown Biasescited by 116
  14. 2016Recurrent Neural Network Grammarscited by 109
  15. 2020On Robustness and Transferability of Convolutional Neural Networkscited by 98
  16. 2019BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performancecited by 89
  17. 2021Contrastive Explanations for Model Interpretabilitycited by 75
  18. 2020Does syntax need to grow on trees? Sources of hierarchical inductive bias in sequence-to-sequence networkscited by 72
  19. 2019Natural Language Understanding with the Quora Question Pairs Datasetcited by 51
  20. 2020Exploring BERT's Sensitivity to Lexical Cues using Tests from Semantic Primingcited by 41
  21. 2021The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Failcited by 38
  22. 2019Studying the Inductive Biases of RNNs with Synthetic Variations of Natural Languagescited by 37
  23. 2019Controlled Crowdsourcing for High-Quality QA-SRL Annotationcited by 34
  24. 2020QADiscourse -- Discourse Relations as QA Pairs: Representation, Crowdsourcing and Baselinescited by 34
  25. 2022Quality Controlled Paraphrase Generationcited by 31
  26. 2020Pretrained Language Model Embryology: The Birth of ALBERTcited by 30
  27. 2018Neural Models for Reasoning over Multiple Mentions using Coreferencecited by 27
  28. 2020Improving Robustness by Augmenting Training Sentences with Predicate-Argument Structurescited by 10
  29. 2020Undersensitivity in Neural Reading Comprehensioncited by 9
  30. 2020Are Some Words Worth More than Others?cited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 1; 2025: 1; 2026: 3.

9 papers.

  1. 2023Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluationcited by 136
  2. 2023Guiding LLM to Fool Itself: Automatically Manipulating Machine Reading Comprehension Shortcut Triggerscited by 10
  3. 2023QAID: Question Answering Inspired Few-shot Intent Detectioncited by 5
  4. 2024On the Effect of (Near) Duplicate Subwords in Language Modellingcited by 5
  5. 2025SeqPE: Transformer with Sequential Position Encodingcited by 2
  6. 2026Names Don't Matter: Symbol-Invariant Transformer for Open-Vocabulary Learningcited by 1
  7. 2026On the Persistent Effects of Lexicality in Large Language Modelscited by 1
  8. 2023Measuring and Improving Attentiveness to Partial Inputs with Counterfactualscited by 0
  9. 2026Memorize Theorems, Not Instances: Probing SFT Generalization through Mathematical Reasoningcited by 0