mlsanityData to 7 Oct 2026

Probing Across Time: What Does RoBERTa Know and When?

arXiv 2104.07885 ↗First posted 16 Apr 2021cs.CLcited by 73

Not on the map. Why?

Abstract (arXiv)

Models of language trained on very large corpora have been demonstrated useful for NLP. As fixed artifacts, they have become the object of intense study, with many researchers "probing" the extent to which linguistic abstractions, factual and commonsense knowledge, and reasoning abilities they acquire and readily demonstrate. Building on this line of work, we consider a new question: for types of knowledge a language model learns, when during (pre)training are they acquired? We plot probing performance across iterations, using RoBERTa as a case study. Among our findings: linguistic knowledge is acquired fast, stably, and robustly across domains. Facts and commonsense are slower and more domain-sensitive. Reasoning abilities are, in general, not stably acquired. As new datasets, pretraining protocols, and probes emerge, we believe that probing-across-time analyses can help researchers understand the complex, intermingled learning that these models undergo and guide us toward more efficient approaches that accomplish necessary learning faster.

Key facts. Cited by 73 papers in the map (data through 7 Oct 2026). Builds on 30 of its 42 references that are in the map. First posted on arXiv 16 Apr 2021.

Builds on

30 of 42 references.

References · 30

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  7. 2018Deep contextualized word representationscited by 2,285
  8. 2019Language Models as Knowledge Bases?cited by 1,922
  9. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  10. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  11. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  12. 2018Neural Network Acceptability Judgmentscited by 952
18 more
  1. 2019Designing and Interpreting Probes with Control Taskscited by 608
  2. 2019Defending Against Neural Fake Newscited by 553
  3. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 553
  4. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
  5. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 413
  6. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
  7. 2019Abductive Commonsense Reasoningcited by 300
  8. 2019Revealing the Dark Secrets of BERTcited by 300
  9. 2018A Simple Method for Commonsense Reasoningcited by 223
  10. 2020Information-Theoretic Probing for Linguistic Structurecited by 204
  11. 2018ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehensioncited by 198
  12. 2019oLMpics -- On what Language Model Pre-training Capturescited by 161
  13. 2020When Do You Need Billions of Words of Pretraining Data?cited by 104
  14. 2019Evaluating Commonsense in Pre-trained Language Modelscited by 95
  15. 2019Visualizing and Understanding the Effectiveness of BERTcited by 67
  16. 2020Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoningcited by 65
  17. 2020Pretrained Language Model Embryology: The Birth of ALBERTcited by 30
  18. 2020Green Algorithms: Quantifying the carbon footprint of computationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 16; 2023: 17; 2024: 15; 2025: 12; 2026: 6.

73 papers.

Sort:

By area:

  1. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  2. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  3. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  4. 2023Language Models Represent Space and Timecited by 306
  5. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 218
  6. 2022A Review on Language Models as Knowledge Basescited by 156
  7. 2024Rho-1: Not All Tokens Are What You Needcited by 132
  8. 2023Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMscited by 121
  9. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  10. 2021The MultiBERTs: BERT Reproductions for Robustness Analysiscited by 89
  11. 2024How Do Large Language Models Acquire Factual Knowledge During Pretraining?cited by 86
  12. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
38 more
  1. 2022Training Trajectories of Language Models Across Scalescited by 80
  2. 2023GeoLLM: Extracting Geospatial Knowledge from Large Language Modelscited by 62
  3. 2022Prompting Language Models for Linguistic Structurecited by 39
  4. 2024Probing the Emergence of Cross-lingual Alignment during LLM Trainingcited by 38
  5. 2021The Grammar-Learning Trajectories of Neural Language Modelscited by 36
  6. 2023Characterizing Learning Curves During Language Model Pre-Training: Learning, Forgetting, and Stabilitycited by 34
  7. 2022Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Modelscited by 32
  8. 2024Goldfish: Monolingual Language Models for 350 Languagescited by 27
  9. 2021Training Dynamics for Text Summarization Modelscited by 25
  10. 2022Neural reality of argument structure constructionscited by 25
  11. 2023Language Model Analysis for Ontology Subsumption Inferencecited by 25
  12. 2023The Life Cycle of Knowledge in Big Language Models: A Surveycited by 23
  13. 2023Subspace Chronicles: How Linguistic Information Emerges, Shifts and Interacts during Language Model Trainingcited by 23
  14. 2024A distributional simplicity bias in the learning dynamics of transformerscited by 23
  15. 2021A Bayesian Framework for Information-Theoretic Probingcited by 22
  16. 2024Explainable Generative AI (GenXAI): A Survey, Conceptualization, and Research Agendacited by 22
  17. 2022Question Generation for Reading Comprehension Assessment by Modeling How and What to Askcited by 20
  18. 2023Does ChatGPT have Theory of Mind?cited by 16
  19. 2024Knowledge Entropy Decay during Language Model Pretraining Hinders New Knowledge Acquisitioncited by 15
  20. 2021Interpreting Language Models Through Knowledge Graph Extractioncited by 14
  21. 2023The future of human-centric eXplainable Artificial Intelligence (XAI) is not post-hoc explanationscited by 12
  22. 2025Dialogue Is Not Enough to Make a Communicative BabyLM (But Neither Is Developmentally Inspired Reinforcement Learning)cited by 10
  23. 2021Is My Model Using The Right Evidence? Systematic Probes for Examining Evidence-Based Tabular Reasoningcited by 9
  24. 2024A Survey on Symbolic Knowledge Distillation of Large Language Modelscited by 9
  25. 2021Does Pre-training Induce Systematic Inference? How Masked Language Models Acquire Commonsense Knowledgecited by 8
  26. 2023Second Language Acquisition of Neural Language Modelscited by 8
  27. 2024Towards More Realistic Extraction Attacks: An Adversarial Perspectivecited by 8
  28. 2024What Do Speech Foundation Models Not Learn About Speech?cited by 8
  29. 2026What do Language Models Learn and When? The Implicit Curriculum Hypothesiscited by 8
  30. 2022SocioProbe: What, When, and Where Language Models Learn about Sociodemographicscited by 7
  31. 2024Phase Transitions in the Output Distribution of Large Language Modelscited by 7
  32. 2025Linguistic Interpretability of Transformer-based Language Models: a systematic reviewcited by 7
  33. 2022A Natural Bias for Language Generation Modelscited by 6
  34. 2024Understanding Token Probability Encoding in Output Embeddingscited by 6
  35. 2025Crosscoding Through Time: Tracking Emergence & Consolidation Of Linguistic Representations Throughout LLM Pretrainingcited by 6
  36. 2022Life after BERT: What do Other Muppets Understand about Language?cited by 4
  37. 2023Competence-Based Analysis of Language Modelscited by 4
  38. 2022Is neural language acquisition similar to natural? A chronological probing studycited by 3

50 most cited of 73 All 73 (JSON)