mlsanityData to 7 Oct 2026

Does Pre-training Induce Systematic Inference? How Masked Language Models Acquire Commonsense Knowledge

arXiv 2112.08583 ↗First posted 16 Dec 2021cs.CLcited by 8

Not on the map. Why?

Abstract (arXiv)

Transformer models pre-trained with a masked-language-modeling objective (e.g., BERT) encode commonsense knowledge as evidenced by behavioral probes; however, the extent to which this knowledge is acquired by systematic inference over the semantics of the pre-training corpora is an open question. To answer this question, we selectively inject verbalized knowledge into the minibatches of a BERT model during pre-training and evaluate how well the model generalizes to supported inferences. We find generalization does not improve over the course of pre-training, suggesting that commonsense knowledge is acquired from surface-level, co-occurrence patterns rather than induced, systematic reasoning.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 34 of its 49 references that are in the map. First posted on arXiv 16 Dec 2021.

Builds on

34 of 49 references.

References · 34

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  5. 2019Language Models as Knowledge Bases?cited by 1,922
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  7. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  8. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  9. 2021Editing Factual Knowledge in Language Modelscited by 548
  10. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  11. 2019COMET: Commonsense Transformers for Automatic Knowledge Graph Constructioncited by 450
  12. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 317
22 more
  1. 2020ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Languagecited by 309
  2. 2021Symbolic Knowledge Distillation: from General Language Models to Commonsense Modelscited by 270
  3. 2020Transformers as Soft Reasoners over Languagecited by 265
  4. 2020COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphscited by 254
  5. 2019WinoGrande: An Adversarial Winograd Schema Challenge at Scalecited by 194
  6. 2021Implicit Representations of Meaning in Neural Language Modelscited by 179
  7. 2019oLMpics -- On what Language Model Pre-training Capturescited by 161
  8. 2020Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both?cited by 147
  9. 2019Evaluating Commonsense in Pre-trained Language Modelscited by 95
  10. 2021Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefscited by 79
  11. 2021Probing Across Time: What Does RoBERTa Know and When?cited by 73
  12. 2019Do Neural Language Representations Learn Physical Commonsense?cited by 62
  13. 2021Frequency Effects on Syntactic Rule Learning in Transformerscited by 59
  14. 2021Provable Limitations of Acquiring Meaning from Ungrounded Form: What Will Future Language Models Understand?cited by 55
  15. 2020TransOMCS: From Linguistic Graphs to Commonsense Knowledgecited by 52
  16. 2020Measuring Systematic Generalization in Neural Proof Generation with Transformerscited by 43
  17. 2021General-Purpose Question-Answering with Macawcited by 40
  18. 2021Do language models learn typicality judgments from text?cited by 32
  19. 2020RICA: Evaluating Robust Inference Capabilities Based on Commonsense Axiomscited by 28
  20. 2021ASER: Towards Large-scale Commonsense Knowledge Acquisition via Higher-order Selectional Preference over Eventualitiescited by 28
  21. 2017Verb Physics: Relative Physical Knowledge of Actions and Objectscited by 26
  22. 2020Leap-Of-Thought: Teaching Pre-Trained Models to Systematically Reason Over Implicit Knowledgecited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 3; 2024: 1; 2025: 2; 2026: 1.

8 papers.

  1. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  2. 2023ChatGPT is a Knowledgeable but Inexperienced Solver: An Investigation of Commonsense Problem in Large Language Modelscited by 63
  3. 2022Language Models as Inductive Reasonerscited by 32
  4. 2025Do Large Language Models Know How Much They Know?cited by 11
  5. 2024What Really is Commonsense Knowledge?cited by 2
  6. 2023Implicit and Explicit Commonsense for Multi-sentence Video Captioningcited by 1
  7. 2025Rewriting History: A Recipe for Interventional Analyses to Study Data Effects on Model Behaviorcited by 0
  8. 2026LLMs Should Incorporate Explicit Mechanisms for Human Empathycited by 0