# Cracking the Contextual Commonsense Code: Understanding Commonsense Reasoning Aptitude of Deep Contextual Representations

arXiv: https://arxiv.org/abs/1910.01157 · HTML: https://mlsanity.com/paper/1910.01157/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Oct 2019.

## Built on by (most cited first, 24 of 24)

- [Measuring and Improving Consistency in Pretrained Language Models](https://mlsanity.com/paper/2102.01017/) (2021)
- [Pre-Trained Models: Past, Present and Future](https://mlsanity.com/paper/2106.07139/) (2021)
- [The World of an Octopus: How Reporting Bias Influences a Language Model's Perception of Color](https://mlsanity.com/paper/2110.08182/) (2021)
- [Relational World Knowledge Representation in Contextual Language Models: A Review](https://mlsanity.com/paper/2104.05837/) (2021)
- [Combining pre-trained language models and structured knowledge](https://mlsanity.com/paper/2101.12294/) (2021)
- [Acquiring and Modelling Abstract Commonsense Knowledge via Conceptualization](https://mlsanity.com/paper/2206.01532/) (2022)
- [COMPS: Conceptual Minimal Pair Sentences for testing Robust Property Knowledge and its Inheritance in Pre-trained Language Models](https://mlsanity.com/paper/2210.01963/) (2022)
- [Probing for Bridging Inference in Transformer Language Models](https://mlsanity.com/paper/2104.09400/) (2021)
- [Does ChatGPT Have a Mind?](https://mlsanity.com/paper/2407.11015/) (2024)
- [Low Anisotropy Sense Retrofitting (LASeR) : Towards Isotropic and Sense Enriched Representations](https://mlsanity.com/paper/2104.10833/) (2021)
- [A Property Induction Framework for Neural Language Models](https://mlsanity.com/paper/2205.06910/) (2022)
- [Evaluating the Construct Validity of Text Embeddings with Application to Survey Questions](https://mlsanity.com/paper/2202.09166/) (2022)
- [On the Tip of the Tongue: Analyzing Conceptual Representation in Large Language Models with Reverse-Dictionary Probe](https://mlsanity.com/paper/2402.14404/) (2024)
- [What do Toothbrushes do in the Kitchen? How Transformers Think our World is Structured](https://mlsanity.com/paper/2204.05673/) (2022)
- [Exploring Multimodal Perception in Large Language Models Through Perceptual Strength Ratings](https://mlsanity.com/paper/2503.06980/) (2025)
- [Identifying the Limits of Cross-Domain Knowledge Transfer for Pretrained Models](https://mlsanity.com/paper/2104.08410/) (2021)
- [Commonsense Knowledge in Word Associations and ConceptNet](https://mlsanity.com/paper/2109.09309/) (2021)
- [On Reality and the Limits of Language Data: Aligning LLMs with Human Norms](https://mlsanity.com/paper/2208.11981/) (2022)
- [Adversarial Transformer Language Models for Contextual Commonsense Inference](https://mlsanity.com/paper/2302.05406/) (2023)
- [Revisiting the Prepositional-Phrase Attachment Problem Using Explicit Commonsense Knowledge](https://mlsanity.com/paper/2102.00924/) (2021)
- [Testing Causal Models of Word Meaning in GPT-3 and -4](https://mlsanity.com/paper/2305.14630/) (2023)
- [Can Language Models Take A Hint? Prompting for Controllable Contextualized Commonsense Inference](https://mlsanity.com/paper/2410.02202/) (2024)
- [Understanding in Artificial Intelligence](https://mlsanity.com/paper/2101.06573/) (2021)
- [Dissecting Paraphrases: The Impact of Prompt Syntax and supplementary Information on Knowledge Retrieval from Pretrained Language Models](https://mlsanity.com/paper/2404.01992/) (2024)

## Abstract (arXiv)

Pretrained deep contextual representations have advanced the state-of-the-art on various commonsense NLP tasks, but we lack a concrete understanding of the capability of these models. Thus, we investigate and challenge several aspects of BERT's commonsense representation abilities. First, we probe BERT's ability to classify various object attributes, demonstrating that BERT shows a strong ability in encoding various commonsense features in its embedding space, but is still deficient in many areas. Next, we show that, by augmenting BERT's pretraining data with additional data related to the deficient attributes, we are able to improve performance on a downstream commonsense reasoning task while using a minimal amount of data. Finally, we develop a method of fine-tuning knowledge graphs embeddings alongside BERT and show the continued importance of explicit knowledge graphs.