# Probing Physical Reasoning with Counter-Commonsense Context

arXiv: https://arxiv.org/abs/2306.02258 · HTML: https://mlsanity.com/paper/2306.02258/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 18 of its 28 references that are in the map.
- First posted on arXiv 4 Jun 2023.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [oLMpics -- On what Language Model Pre-training Captures](https://mlsanity.com/paper/1912.13283/) (2019)
- [Do Language Embeddings Capture Scales?](https://mlsanity.com/paper/2010.05345/) (2020)
- [Do Neural Language Representations Learn Physical Commonsense?](https://mlsanity.com/paper/1908.02899/) (2019)
- [PROST: Physical Reasoning of Objects through Space and Time](https://mlsanity.com/paper/2106.03634/) (2021)
- [PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D World](https://mlsanity.com/paper/2106.00188/) (2021)
- [Things not Written in Text: Exploring Spatial Commonsense from Visual Signals](https://mlsanity.com/paper/2203.08075/) (2022)
- [Dimensions of Commonsense Knowledge](https://mlsanity.com/paper/2101.04640/) (2021)
- [Verb Physics: Relative Physical Knowledge of Actions and Objects](https://mlsanity.com/paper/1706.03799/) (2017)
- [How Large Are Lions? Inducing Distributions over Quantitative Attributes](https://mlsanity.com/paper/1906.01327/) (2019)
- [PACS: A Dataset for Physical Audiovisual CommonSense Reasoning](https://mlsanity.com/paper/2203.11130/) (2022)
- [Penguins Don't Fly: Reasoning about Generics through Instantiations and Exceptions](https://mlsanity.com/paper/2205.11658/) (2022)

## Built on by (most cited first, 4 of 4)

- [Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks](https://mlsanity.com/paper/2307.02477/) (2023)
- [Integration of cognitive tasks into artificial general intelligence test for large models](https://mlsanity.com/paper/2402.02547/) (2024)
- [ACCORD: Closing the Commonsense Measurability Gap](https://mlsanity.com/paper/2406.02804/) (2024)
- [Finding Sense in Nonsense with Generated Contexts: Perspectives from Humans and Language Models](https://mlsanity.com/paper/2602.11699/) (2026)

## Abstract (arXiv)

In this study, we create a CConS (Counter-commonsense Contextual Size comparison) dataset to investigate how physical commonsense affects the contextualized size comparison task; the proposed dataset consists of both contexts that fit physical commonsense and those that do not. This dataset tests the ability of language models to predict the size relationship between objects under various contexts generated from our curated noun list and templates. We measure the ability of several masked language models and generative models. The results show that while large language models can use prepositions such as ``in'' and ``into'' in the provided context to infer size relationships, they fail to use verbs and thus make incorrect judgments led by their prior physical commonsense.