# Analyzing Commonsense Emergence in Few-shot Knowledge Models

arXiv: https://arxiv.org/abs/2101.00297 · HTML: https://mlsanity.com/paper/2101.00297/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 46 references that are in the map.
- First posted on arXiv 1 Jan 2021.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [COMET: Commonsense Transformers for Automatic Knowledge Graph Construction](https://mlsanity.com/paper/1906.05317/) (2019)
- [Abductive Commonsense Reasoning](https://mlsanity.com/paper/1908.05739/) (2019)
- [Transformers as Soft Reasoners over Language](https://mlsanity.com/paper/2002.05867/) (2020)
- [COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphs](https://mlsanity.com/paper/2010.05953/) (2020)
- [Commonsense Knowledge Mining from Pretrained Models](https://mlsanity.com/paper/1909.00505/) (2019)
- [Evaluating Commonsense in Pre-trained Language Models](https://mlsanity.com/paper/1911.11931/) (2019)
- [Commonsense Knowledge Base Completion with Structural and Semantic Context](https://mlsanity.com/paper/1910.02915/) (2019)
- [Connecting the Dots: A Knowledgeable Path Generator for Commonsense Question Answering](https://mlsanity.com/paper/2005.00691/) (2020)
- [Inductive Learning on Commonsense Knowledge Graph Completion](https://mlsanity.com/paper/2009.09263/) (2020)

## Built on by (most cited first, 30 of 34)

- [Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space](https://mlsanity.com/paper/2203.14680/) (2022)
- [A Review on Language Models as Knowledge Bases](https://mlsanity.com/paper/2204.06031/) (2022)
- [Model Editing Harms General Abilities of Large Language Models: Regularization to the Rescue](https://mlsanity.com/paper/2401.04700/) (2024)
- [Language Models Implement Simple Word2Vec-style Vector Arithmetic](https://mlsanity.com/paper/2305.16130/) (2023)
- [How Do Large Language Models Acquire Factual Knowledge During Pretraining?](https://mlsanity.com/paper/2406.11813/) (2024)
- [Neuro-Symbolic AI in 2024: A Systematic Review](https://mlsanity.com/paper/2501.05435/) (2025)
- [An In-depth Survey of Large Language Model-based Artificial Intelligence Agents](https://mlsanity.com/paper/2309.14365/) (2023)
- [Are Large Language Models Geospatially Knowledgeable?](https://mlsanity.com/paper/2310.13002/) (2023)
- [Perturbation-Restrained Sequential Model Editing](https://mlsanity.com/paper/2405.16821/) (2024)
- [Language Model Analysis for Ontology Subsumption Inference](https://mlsanity.com/paper/2302.06761/) (2023)
- [RECKONING: Reasoning through Dynamic Knowledge Encoding](https://mlsanity.com/paper/2305.06349/) (2023)
- [Discovering Knowledge-Critical Subnetworks in Pretrained Language Models](https://mlsanity.com/paper/2310.03084/) (2023)
- [Data Mixing Can Induce Phase Transitions in Knowledge Acquisition](https://mlsanity.com/paper/2505.18091/) (2025)
- [StereoKG: Data-Driven Knowledge Graph Construction for Cultural Knowledge and Stereotypes](https://mlsanity.com/paper/2205.14036/) (2022)
- [Understanding Finetuning for Factual Knowledge Extraction from Language Models](https://mlsanity.com/paper/2301.11293/) (2023)
- [Diversify and Conquer: Diversity-Centric Data Selection with Iterative Refinement](https://mlsanity.com/paper/2409.11378/) (2024)
- [ComFact: A Benchmark for Linking Contextual Commonsense Knowledge](https://mlsanity.com/paper/2210.12678/) (2022)
- [Inferring Implicit Relations in Complex Questions with Language Models](https://mlsanity.com/paper/2204.13778/) (2022)
- [Exploiting Edited Large Language Models as General Scientific Optimizers](https://mlsanity.com/paper/2503.09620/) (2025)
- [VIPHY: Probing "Visible" Physical Commonsense Knowledge](https://mlsanity.com/paper/2209.07000/) (2022)
- [kogito: A Commonsense Knowledge Inference Toolkit](https://mlsanity.com/paper/2211.08451/) (2022)
- [TSGP: Two-Stage Generative Prompting for Unsupervised Commonsense Question Answering](https://mlsanity.com/paper/2211.13515/) (2022)
- [The Impact of Generative Artificial Intelligence on Ideation and the performance of Innovation Teams (Preprint)](https://mlsanity.com/paper/2410.18357/) (2024)
- [DiffuCOMET: Contextual Commonsense Knowledge Diffusion](https://mlsanity.com/paper/2402.17011/) (2024)
- [Bridging Linguistic Structure and Mechanistic Interpretability for Conceptual Interpretation in Language Models](https://mlsanity.com/paper/2408.11827/) (2024)
- [Counteracts: Testing Stereotypical Representation in Pre-trained Language Models](https://mlsanity.com/paper/2301.04347/) (2023)
- [On the Relationship between Skill Neurons and Robustness in Prompt Tuning](https://mlsanity.com/paper/2309.12263/) (2023)
- [Can Language Models Take A Hint? Prompting for Controllable Contextualized Commonsense Inference](https://mlsanity.com/paper/2410.02202/) (2024)
- [Promote, Suppress, Iterate: How Language Models Answer One-to-Many Factual Queries](https://mlsanity.com/paper/2502.20475/) (2025)
- [Matching Tasks to Objectives: Fine-Tuning and Prompt-Tuning Strategies for Encoder-Decoder Pre-trained Language Models](https://mlsanity.com/paper/2606.24841/) (2026)

## Abstract (arXiv)

Recently, commonsense knowledge models - pretrained language models (LM) fine-tuned on knowledge graph (KG) tuples - showed that considerable amounts of commonsense knowledge can be encoded in the parameters of large language models. However, as parallel studies show that LMs are poor hypothesizers of declarative commonsense relationships on their own, it remains unclear whether this knowledge is learned during pretraining or from fine-tuning on KG examples. To investigate this question, we train commonsense knowledge models in few-shot settings to study the emergence of their commonsense representation abilities. Our results show that commonsense knowledge models can rapidly adapt from limited examples, indicating that KG fine-tuning serves to learn an interface to encoded knowledge learned during pretraining. Importantly, our analysis of absolute, angular, and distributional parameter changes during few-shot fine-tuning provides novel insights into how this interface is learned.