# On Commonsense Cues in BERT for Solving Commonsense Tasks

arXiv: https://arxiv.org/abs/2008.03945 · HTML: https://mlsanity.com/paper/2008.03945/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Aug 2020.

## Built on by (most cited first, 13 of 13)

- [A Review on Language Models as Knowledge Bases](https://mlsanity.com/paper/2204.06031/) (2022)
- [ChatGPT is a Knowledgeable but Inexperienced Solver: An Investigation of Commonsense Problem in Large Language Models](https://mlsanity.com/paper/2303.16421/) (2023)
- [DKPLM: Decomposable Knowledge-enhanced Pre-trained Language Model for Natural Language Understanding](https://mlsanity.com/paper/2112.01047/) (2021)
- [Can Pre-trained Language Models Interpret Similes as Smart as Human?](https://mlsanity.com/paper/2203.08452/) (2022)
- [A Close Look at Decomposition-based XAI-Methods for Transformer Language Models](https://mlsanity.com/paper/2502.15886/) (2025)
- [Revisiting and Advancing Chinese Natural Language Understanding with Accelerated Heterogeneous Knowledge Pre-training](https://mlsanity.com/paper/2210.05287/) (2022)
- [Unsupervised Explanation Generation via Correct Instantiations](https://mlsanity.com/paper/2211.11160/) (2022)
- [KnowPrefix-Tuning: A Two-Stage Prefix-Tuning Framework for Knowledge-Grounded Dialogue Generation](https://mlsanity.com/paper/2306.15430/) (2023)
- [A Survey of Knowledge Enhanced Pre-trained Models](https://mlsanity.com/paper/2110.00269/) (2021)
- [UniToMBench: Integrating Perspective-Taking to Improve Theory of Mind in LLMs](https://mlsanity.com/paper/2506.09450/) (2025)
- [CommonsenseVIS: Visualizing and Understanding Commonsense Reasoning Capabilities of Natural Language Models](https://mlsanity.com/paper/2307.12382/) (2023)
- [Do Transformers know symbolic rules, and would we know if they did?](https://mlsanity.com/paper/2203.00162/) (2022)
- [TRELM: Towards Robust and Efficient Pre-training for Knowledge-Enhanced Language Models](https://mlsanity.com/paper/2403.11203/) (2024)

## Abstract (arXiv)

BERT has been used for solving commonsense tasks such as CommonsenseQA. While prior research has found that BERT does contain commonsense information to some extent, there has been work showing that pre-trained models can rely on spurious associations (e.g., data bias) rather than key cues in solving sentiment classification and other problems. We quantitatively investigate the presence of structural commonsense cues in BERT when solving commonsense tasks, and the importance of such cues for the model prediction. Using two different measures, we find that BERT does use relevant knowledge for solving the task, and the presence of commonsense knowledge is positively correlated to the model accuracy.