# BERTnesia: Investigating the capture and forgetting of knowledge in BERT

arXiv: https://arxiv.org/abs/2106.02902 · HTML: https://mlsanity.com/paper/2106.02902/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 53 of its 82 references that are in the map.
- First posted on arXiv 5 Jun 2021.

## Builds on (most cited first, 30 of 53)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020)
- [Passage Re-ranking with BERT](https://mlsanity.com/paper/1901.04085/) (2019)
- [Knowledge Neurons in Pretrained Transformers](https://mlsanity.com/paper/2104.08696/) (2021)
- [KILT: a Benchmark for Knowledge Intensive Language Tasks](https://mlsanity.com/paper/2009.02252/) (2020)
- [What do you learn from context? Probing for sentence structure in contextualized word representations](https://mlsanity.com/paper/1905.06316/) (2019)
- [Linguistic Knowledge and Transferability of Contextual Representations](https://mlsanity.com/paper/1903.08855/) (2019)
- [Measuring and Improving Consistency in Pretrained Language Models](https://mlsanity.com/paper/2102.01017/) (2021)
- [DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference](https://mlsanity.com/paper/2004.12993/) (2020)
- [K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters](https://mlsanity.com/paper/2002.01808/) (2020)
- [K-BERT: Enabling Language Representation with Knowledge Graph](https://mlsanity.com/paper/1909.07606/) (2019)
- [ERNIE: Enhanced Representation through Knowledge Integration](https://mlsanity.com/paper/1904.09223/) (2019)
- [Revealing the Dark Secrets of BERT](https://mlsanity.com/paper/1908.08593/) (2019)
- [Knowledge Enhanced Contextual Word Representations](https://mlsanity.com/paper/1909.04164/) (2019)
- [Factual Probing Is [MASK]: Learning vs. Learning to Recall](https://mlsanity.com/paper/2104.05240/) (2021)

## Built on by (most cited first, 30 of 40)

- [Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space](https://mlsanity.com/paper/2203.14680/) (2022)
- [Dissecting Recall of Factual Associations in Auto-Regressive Language Models](https://mlsanity.com/paper/2304.14767/) (2023)
- [Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models](https://mlsanity.com/paper/2401.06102/) (2024)
- [A Review on Language Models as Knowledge Bases](https://mlsanity.com/paper/2204.06031/) (2022)
- [Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patching](https://mlsanity.com/paper/2311.17030/) (2023)
- [Understanding Transformer Memorization Recall Through Idioms](https://mlsanity.com/paper/2210.03588/) (2022)
- [Mechanistic Understanding and Mitigation of Language Model Non-Factual Hallucinations](https://mlsanity.com/paper/2403.18167/) (2024)
- [Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Models](https://mlsanity.com/paper/2310.16570/) (2023)
- [DeCoRe: Decoding by Contrasting Retrieval Heads to Mitigate Hallucinations](https://mlsanity.com/paper/2410.18860/) (2024)
- [A Glitch in the Matrix? Locating and Detecting Language Model Grounding with Fakepedia](https://mlsanity.com/paper/2312.02073/) (2023)
- [The Life Cycle of Knowledge in Big Language Models: A Survey](https://mlsanity.com/paper/2303.07616/) (2023)
- [Discovering Knowledge-Critical Subnetworks in Pretrained Language Models](https://mlsanity.com/paper/2310.03084/) (2023)
- [Unveiling Factual Recall Behaviors of Large Language Models through Knowledge Neurons](https://mlsanity.com/paper/2408.03247/) (2024)
- [Parameter-Efficient Tuning Makes a Good Classification Head](https://mlsanity.com/paper/2210.16771/) (2022)
- [Why do universal adversarial attacks work on large language models?: Geometry might be the answer](https://mlsanity.com/paper/2309.00254/) (2023)
- [Understanding Finetuning for Factual Knowledge Extraction from Language Models](https://mlsanity.com/paper/2301.11293/) (2023)
- [Do Large Language Models Know How Much They Know?](https://mlsanity.com/paper/2502.19573/) (2025)
- [Beyond Distributional Hypothesis: Let Language Models Learn Meaning-Text Correspondence](https://mlsanity.com/paper/2205.03815/) (2022)
- [Depth-Wise Attention (DWAtt): A Layer Fusion Method for Data-Efficient Classification](https://mlsanity.com/paper/2209.15168/) (2022)
- [The Cost of Compression: Investigating the Impact of Compression on Parametric Knowledge in Language Models](https://mlsanity.com/paper/2312.00960/) (2023)
- [Finding Structural Knowledge in Multimodal-BERT](https://mlsanity.com/paper/2203.09306/) (2022)
- [Defining Knowledge: Bridging Epistemology and Large Language Models](https://mlsanity.com/paper/2410.02499/) (2024)
- [Unsupervised Improvement of Factual Knowledge in Language Models](https://mlsanity.com/paper/2304.01597/) (2023)
- [Can Large Language Models Change User Preference Adversarially?](https://mlsanity.com/paper/2302.10291/) (2023)
- [Improving Language Models Meaning Understanding and Consistency by Learning Conceptual Roles from Dictionary](https://mlsanity.com/paper/2310.15541/) (2023)
- [Decouple knowledge from parameters for plug-and-play language modeling](https://mlsanity.com/paper/2305.11564/) (2023)
- [Superscopes: Amplifying Internal Feature Representations for Language Model Interpretation](https://mlsanity.com/paper/2503.02078/) (2025)
- [Efficient Knowledge Probing of Large Language Models by Adapting Pre-trained Embeddings](https://mlsanity.com/paper/2508.06030/) (2025)
- [LoKI: Low-damage Knowledge Implanting of Large Language Models](https://mlsanity.com/paper/2505.22120/) (2025)
- [Consistency and Coherence from Points of Contextual Similarity](https://mlsanity.com/paper/2112.11638/) (2021)

## Abstract (arXiv)

Probing complex language models has recently revealed several insights into linguistic and semantic patterns found in the learned representations. In this article, we probe BERT specifically to understand and measure the relational knowledge it captures in its parametric memory. While probing for linguistic understanding is commonly applied to all layers of BERT as well as fine-tuned models, this has not been done for factual knowledge. We utilize existing knowledge base completion tasks (LAMA) to probe every layer of pre-trained as well as fine-tuned BERT models(ranking, question answering, NER). Our findings show that knowledge is not just contained in BERT's final layers. Intermediate layers contribute a significant amount (17-60%) to the total knowledge found. Probing intermediate layers also reveals how different types of knowledge emerge at varying rates. When BERT is fine-tuned, relational knowledge is forgotten. The extent of forgetting is impacted by the fine-tuning objective and the training data. We found that ranking models forget the least and retain more knowledge in their final layer compared to masked language modeling and question-answering. However, masked language modeling performed the best at acquiring new knowledge from the training data. When it comes to learning facts, we found that capacity and fact density are key factors. We hope this initial work will spur further research into understanding the parametric memory of language models and the effect of training objectives on factual knowledge. The code to repeat the experiments is publicly available on GitHub.