# Probing Across Time: What Does RoBERTa Know and When?

arXiv: https://arxiv.org/abs/2104.07885 · HTML: https://mlsanity.com/paper/2104.07885/

## Key facts

- Cited by 73 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 42 references that are in the map.
- First posted on arXiv 16 Apr 2021.

## Builds on (most cited first, 30 of 30)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018)
- [Designing and Interpreting Probes with Control Tasks](https://mlsanity.com/paper/1909.03368/) (2019)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [Defending Against Neural Fake News](https://mlsanity.com/paper/1905.12616/) (2019)
- [What do you learn from context? Probing for sentence structure in contextualized word representations](https://mlsanity.com/paper/1905.06316/) (2019)
- [A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories](https://mlsanity.com/paper/1604.01696/) (2016)
- [Linguistic Knowledge and Transferability of Contextual Representations](https://mlsanity.com/paper/1903.08855/) (2019)
- [Abductive Commonsense Reasoning](https://mlsanity.com/paper/1908.05739/) (2019)
- [Revealing the Dark Secrets of BERT](https://mlsanity.com/paper/1908.08593/) (2019)
- [A Simple Method for Commonsense Reasoning](https://mlsanity.com/paper/1806.02847/) (2018)
- [Information-Theoretic Probing for Linguistic Structure](https://mlsanity.com/paper/2004.03061/) (2020)
- [ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension](https://mlsanity.com/paper/1810.12885/) (2018)
- [oLMpics -- On what Language Model Pre-training Captures](https://mlsanity.com/paper/1912.13283/) (2019)
- [When Do You Need Billions of Words of Pretraining Data?](https://mlsanity.com/paper/2011.04946/) (2020)
- [Evaluating Commonsense in Pre-trained Language Models](https://mlsanity.com/paper/1911.11931/) (2019)
- [Visualizing and Understanding the Effectiveness of BERT](https://mlsanity.com/paper/1908.05620/) (2019)
- [Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning](https://mlsanity.com/paper/2010.05906/) (2020)
- [Pretrained Language Model Embryology: The Birth of ALBERT](https://mlsanity.com/paper/2010.02480/) (2020)
- [Green Algorithms: Quantifying the carbon footprint of computation](https://mlsanity.com/paper/2007.07610/) (2020)

## Built on by (most cited first, 30 of 73)

- [Mechanistic Interpretability for AI Safety -- A Review](https://mlsanity.com/paper/2404.14082/) (2024)
- [How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources](https://mlsanity.com/paper/2306.04751/) (2023)
- [Finding Neurons in a Haystack: Case Studies with Sparse Probing](https://mlsanity.com/paper/2305.01610/) (2023)
- [Language Models Represent Space and Time](https://mlsanity.com/paper/2310.02207/) (2023)
- [Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models](https://mlsanity.com/paper/2205.10770/) (2022)
- [A Review on Language Models as Knowledge Bases](https://mlsanity.com/paper/2204.06031/) (2022)
- [Rho-1: Not All Tokens Are What You Need](https://mlsanity.com/paper/2404.07965/) (2024)
- [Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs](https://mlsanity.com/paper/2309.07311/) (2023)
- [The MultiBERTs: BERT Reproductions for Robustness Analysis](https://mlsanity.com/paper/2106.16163/) (2021)
- [Language Model Behavior: A Comprehensive Survey](https://mlsanity.com/paper/2303.11504/) (2023)
- [How Do Large Language Models Acquire Factual Knowledge During Pretraining?](https://mlsanity.com/paper/2406.11813/) (2024)
- [Training Trajectories of Language Models Across Scales](https://mlsanity.com/paper/2212.09803/) (2022)
- [Cramming: Training a Language Model on a Single GPU in One Day](https://mlsanity.com/paper/2212.14034/) (2022)
- [GeoLLM: Extracting Geospatial Knowledge from Large Language Models](https://mlsanity.com/paper/2310.06213/) (2023)
- [Prompting Language Models for Linguistic Structure](https://mlsanity.com/paper/2211.07830/) (2022)
- [Probing the Emergence of Cross-lingual Alignment during LLM Training](https://mlsanity.com/paper/2406.13229/) (2024)
- [The Grammar-Learning Trajectories of Neural Language Models](https://mlsanity.com/paper/2109.06096/) (2021)
- [Characterizing Learning Curves During Language Model Pre-Training: Learning, Forgetting, and Stability](https://mlsanity.com/paper/2308.15419/) (2023)
- [Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Models](https://mlsanity.com/paper/2205.11758/) (2022)
- [Goldfish: Monolingual Language Models for 350 Languages](https://mlsanity.com/paper/2408.10441/) (2024)
- [Training Dynamics for Text Summarization Models](https://mlsanity.com/paper/2110.08370/) (2021)
- [Neural reality of argument structure constructions](https://mlsanity.com/paper/2202.12246/) (2022)
- [Language Model Analysis for Ontology Subsumption Inference](https://mlsanity.com/paper/2302.06761/) (2023)
- [A distributional simplicity bias in the learning dynamics of transformers](https://mlsanity.com/paper/2410.19637/) (2024)
- [The Life Cycle of Knowledge in Big Language Models: A Survey](https://mlsanity.com/paper/2303.07616/) (2023)
- [Subspace Chronicles: How Linguistic Information Emerges, Shifts and Interacts during Language Model Training](https://mlsanity.com/paper/2310.16484/) (2023)
- [A Bayesian Framework for Information-Theoretic Probing](https://mlsanity.com/paper/2109.03853/) (2021)
- [Explainable Generative AI (GenXAI): A Survey, Conceptualization, and Research Agenda](https://mlsanity.com/paper/2404.09554/) (2024)
- [Question Generation for Reading Comprehension Assessment by Modeling How and What to Ask](https://mlsanity.com/paper/2204.02908/) (2022)
- [Does ChatGPT have Theory of Mind?](https://mlsanity.com/paper/2305.14020/) (2023)

## Abstract (arXiv)

Models of language trained on very large corpora have been demonstrated useful for NLP. As fixed artifacts, they have become the object of intense study, with many researchers "probing" the extent to which linguistic abstractions, factual and commonsense knowledge, and reasoning abilities they acquire and readily demonstrate. Building on this line of work, we consider a new question: for types of knowledge a language model learns, when during (pre)training are they acquired? We plot probing performance across iterations, using RoBERTa as a case study. Among our findings: linguistic knowledge is acquired fast, stably, and robustly across domains. Facts and commonsense are slower and more domain-sensitive. Reasoning abilities are, in general, not stably acquired. As new datasets, pretraining protocols, and probes emerge, we believe that probing-across-time analyses can help researchers understand the complex, intermingled learning that these models undergo and guide us toward more efficient approaches that accomplish necessary learning faster.