mlsanityData to 7 Oct 2026

PreCog: Exploring the Relation between Memorization and Performance in Pre-trained Language Models

arXiv 2305.04673 ↗First posted 8 May 2023cs.CL cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Pre-trained Language Models such as BERT are impressive machines with the ability to memorize, possibly generalized learning examples. We present here a small, focused contribution to the analysis of the interplay between memorization and performance of BERT in downstream tasks. We propose PreCog, a measure for evaluating memorization from pre-training, and we analyze its correlation with the BERT's performance. Our experiments show that highly memorized examples are better classified, suggesting memorization is an essential key to success for BERT.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 25 of its 42 references that are in the map. First posted on arXiv 8 May 2023.

Builds on

25 of 42 references.

References · 25

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  8. 2018Deep contextualized word representationscited by 2,285
  9. 2020Extracting Training Data from Large Language Modelscited by 2,144
  10. 2019Language Models as Knowledge Bases?cited by 1,922
  11. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  12. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
13 more
  1. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  2. 2018Neural Network Acceptability Judgmentscited by 952
  3. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  4. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  5. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 669
  6. 2020Pretrained Transformers Improve Out-of-Distribution Robustnesscited by 293
  7. 2021Counterfactual Memorization in Neural Language Modelscited by 155
  8. 2021Lifelong Pretraining: Continually Adapting Language Models to Emerging Corporacited by 114
  9. 2020What Happens To BERT Embeddings During Fine-tuning?cited by 113
  10. 2019Natural Language Understanding with the Quora Question Pairs Datasetcited by 51
  11. 2020Understanding Unintended Memorization in Federated Learningcited by 36
  12. 2021Memorisation versus Generalisation in Pre-trained Language Modelscited by 32
  13. 2019Forget Me Not: Reducing Catastrophic Forgetting for Domain Adaptation in Reading Comprehensioncited by 27

Built on by

Cited by year

Citing papers per year: 2024: 2.

2 papers.

  1. 2024Clinical Insights: A Comprehensive Review of Language Models in Medicinecited by 6
  2. 2024Do LLMs Dream of Ontologies?cited by 3