mlsanityData to 14 Oct 2026

Memorisation versus Generalisation in Pre-trained Language Models

arXiv 2105.00828 ↗First posted 16 Apr 2021cs.CL cs.LGcited by 32

Not on the map. Why?

Abstract (arXiv)

State-of-the-art pre-trained language models have been shown to memorise facts and perform well with limited amounts of training data. To gain a better understanding of how these models learn, we study their generalisation and memorisation capabilities in noisy and low-resource scenarios. We find that the training of these models is almost unaffected by label noise and that it is possible to reach near-optimal results even on extremely noisy datasets. However, our experiments also show that they mainly learn from high-frequency patterns and largely fail when tested on low-resource tasks such as few-shot learning and rare entity recognition. To mitigate such limitations, we propose an extension based on prototypical networks that improves performance in low-resource named entity recognition tasks.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Builds on 26 of its 37 references that are in the map. First posted on arXiv 16 Apr 2021.

Builds on

26 of 37 references.

References · 26

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2017Prototypical Networks for Few-shot Learningcited by 3,015
  6. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  7. 2018Deep contextualized word representationscited by 2,290
  8. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  9. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  10. 2019Language Models as Knowledge Bases?cited by 1,936
  11. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  12. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
14 more
  1. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  2. 2017A Closer Look at Memorization in Deep Networkscited by 942
  3. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 864
  4. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  5. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 678
  6. 2016Neural Architectures for Named Entity Recognitioncited by 550
  7. 2020Early-Learning Regularization Prevents Memorization of Noisy Labelscited by 416
  8. 2020Pretrained Transformers Improve Out-of-Distribution Robustnesscited by 294
  9. 2019Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networkscited by 159
  10. 2020An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Modelscited by 134
  11. 2019CrossWeigh: Training Named Entity Tagger from Imperfect Annotationscited by 60
  12. 2019Cloze-driven Pretraining of Self-attention Networkscited by 41
  13. 2017Generalisation in Named Entity Recognition: A Quantitative Analysiscited by 25
  14. 2020Noisy Text Data: Achilles' Heel of BERTcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 9; 2023: 13; 2024: 6; 2025: 1; 2026: 3.

32 papers.

Sort:

By area:

  1. 2023A Comprehensive Overview of Large Language Modelscited by 648
  2. 2024RAFT: Adapting Language Model to Domain Specific RAGcited by 262
  3. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 230
  4. 2022Does CLIP Bind Concepts? Probing Compositionality in Large Image Modelscited by 102
  5. 2023DyVal: Dynamic Evaluation of Large Language Models for Reasoning Taskscited by 101
  6. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 98
  7. 2024Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristicscited by 97
  8. 2024How Do Large Language Models Acquire Factual Knowledge During Pretraining?cited by 90
  9. 2022Simplicity Bias in Transformers and their Ability to Learn Sparse Boolean Functionscited by 70
  10. 2022Understanding Transformer Memorization Recall Through Idiomscited by 62
  11. 2025How much do language models memorize?cited by 61
  12. 2022Memorization in NLP Fine-tuning Methodscited by 45
20 more
  1. 2022Decoupling Knowledge from Memorization: Retrieval-augmented Prompt Learningcited by 36
  2. 2022A Survey of Methods for Addressing Class Imbalance in Deep-Learning Based Natural Language Processingcited by 21
  3. 2023Koala: An Index for Quantifying Overlaps with Pre-training Corporacited by 19
  4. 2023Retentive or Forgetful? Diving into the Knowledge Memorizing Mechanism of Language Modelscited by 18
  5. 2024Quantifying Generalization Complexity for Large Language Modelscited by 18
  6. 2022Finding Memo: Extractive Memorization in Constrained Sequence Generation Taskscited by 12
  7. 2023CoTEVer: Chain of Thought Prompting Annotation Toolkit for Explanation Verificationcited by 11
  8. 2023Weaker Than You Think: A Critical Look at Weakly Supervised Learningcited by 8
  9. 2024Automatic Question-Answer Generation for Long-Tail Knowledgecited by 8
  10. 2023A Human Subject Study of Named Entity Recognition (NER) in Conversational Music Recommendation Queriescited by 6
  11. 2022On the Security Vulnerabilities of Text-to-SQL Modelscited by 3
  12. 2023Pop Quiz! Do Pre-trained Code Models Possess Knowledge of Correct API Names?cited by 3
  13. 2023PreCog: Exploring the Relation between Memorization and Performance in Pre-trained Language Modelscited by 2
  14. 2023A State-Vector Framework for Dataset Effectscited by 1
  15. 2023Integrating curation into scientific publishing to train AI modelscited by 1
  16. 2024NoisyAG-News: A Benchmark for Addressing Instance-Dependent Noise in Text Classificationcited by 1
  17. 2023Large-scale investigation of weakly-supervised deep learning for the fine-grained semantic indexing of biomedical literaturecited by 0
  18. 2026Query Suggestion for Retrieval-Augmented Generation via Dynamic In-Context Learningcited by 0
  19. 2026Learning to Detect Language Model Training Data via Active Reconstructioncited by 0
  20. 2026Analyzing the Effect of Noise in LLM Fine-tuningcited by 0