# Do Neural Language Representations Learn Physical Commonsense?

arXiv: https://arxiv.org/abs/1908.02899 · HTML: https://mlsanity.com/paper/1908.02899/

## Key facts

- Cited by 62 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Aug 2019.

## Built on by (most cited first, 30 of 62)

- [Measuring and Improving Consistency in Pretrained Language Models](https://mlsanity.com/paper/2102.01017/) (2021)
- [Pre-Trained Models: Past, Present and Future](https://mlsanity.com/paper/2106.07139/) (2021)
- [Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Color](https://mlsanity.com/paper/2109.06129/) (2021)
- [Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases](https://mlsanity.com/paper/2106.09231/) (2021)
- [Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Images](https://mlsanity.com/paper/2303.07274/) (2023)
- [BERTnesia: Investigating the capture and forgetting of knowledge in BERT](https://mlsanity.com/paper/2106.02902/) (2021)
- [Commonsense Knowledge Reasoning and Generation with Pre-trained Language Models: A Survey](https://mlsanity.com/paper/2201.12438/) (2022)
- [Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema](https://mlsanity.com/paper/2104.08161/) (2021)
- [Beyond Surface Statistics: Scene Representations in a Latent Diffusion Model](https://mlsanity.com/paper/2306.05720/) (2023)
- [Does Vision-and-Language Pretraining Improve Lexical Grounding?](https://mlsanity.com/paper/2109.10246/) (2021)
- [The Life Cycle of Knowledge in Big Language Models: A Survey](https://mlsanity.com/paper/2303.07616/) (2023)
- [IRFL: Image Recognition of Figurative Language](https://mlsanity.com/paper/2303.15445/) (2023)
- [VASR: Visual Analogies of Situation Recognition](https://mlsanity.com/paper/2212.04542/) (2022)
- [WinoGAViL: Gamified Association Benchmark to Challenge Vision-and-Language Models](https://mlsanity.com/paper/2207.12576/) (2022)
- [Is BERT Blind? Exploring the Effect of Vision-and-Language Pretraining on Visual Language Understanding](https://mlsanity.com/paper/2303.12513/) (2023)
- [Tiered Reasoning for Intuitive Physics: Toward Verifiable Commonsense Language Understanding](https://mlsanity.com/paper/2109.04947/) (2021)
- [Can Language Models Understand Physical Concepts?](https://mlsanity.com/paper/2305.14057/) (2023)
- [Do Language Models Know the Way to Rome?](https://mlsanity.com/paper/2109.07971/) (2021)
- [PACS: A Dataset for Physical Audiovisual CommonSense Reasoning](https://mlsanity.com/paper/2203.11130/) (2022)
- [Retentive or Forgetful? Diving into the Knowledge Memorizing Mechanism of Language Models](https://mlsanity.com/paper/2305.09144/) (2023)
- [Acquiring and Modelling Abstract Commonsense Knowledge via Conceptualization](https://mlsanity.com/paper/2206.01532/) (2022)
- [Distilling Relation Embeddings from Pre-trained Language Models](https://mlsanity.com/paper/2110.15705/) (2021)
- [COMPS: Conceptual Minimal Pair Sentences for testing Robust Property Knowledge and its Inheritance in Pre-trained Language Models](https://mlsanity.com/paper/2210.01963/) (2022)
- [Does ChatGPT Have a Mind?](https://mlsanity.com/paper/2407.11015/) (2024)
- [Disentangled Counterfactual Learning for Physical Audiovisual Commonsense Reasoning](https://mlsanity.com/paper/2310.19559/) (2023)
- [Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scale](https://mlsanity.com/paper/2510.24963/) (2025)
- [ALL Dolphins Are Intelligent and SOME Are Friendly: Probing BERT for Nouns' Semantic Properties and their Prototypicality](https://mlsanity.com/paper/2110.06376/) (2021)
- [Probing Pre-Trained Language Models for Disease Knowledge](https://mlsanity.com/paper/2106.07285/) (2021)
- [Language Models for Lexical Inference in Context](https://mlsanity.com/paper/2102.05331/) (2021)
- [A Property Induction Framework for Neural Language Models](https://mlsanity.com/paper/2205.06910/) (2022)

## Abstract (arXiv)

Humans understand language based on the rich background knowledge about how the physical world works, which in turn allows us to reason about the physical world through language. In addition to the properties of objects (e.g., boats require fuel) and their affordances, i.e., the actions that are applicable to them (e.g., boats can be driven), we can also reason about if-then inferences between what properties of objects imply the kind of actions that are applicable to them (e.g., that if we can drive something then it likely requires fuel). In this paper, we investigate the extent to which state-of-the-art neural language representations, trained on a vast amount of natural language text, demonstrate physical commonsense reasoning. While recent advancements of neural language models have demonstrated strong performance on various types of natural language inference tasks, our study based on a dataset of over 200k newly collected annotations suggests that neural language representations still only learn associations that are explicitly written down.