# Attention Is (not) All You Need for Commonsense Reasoning

arXiv: https://arxiv.org/abs/1905.13497 · HTML: https://mlsanity.com/paper/1905.13497/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 May 2019.

## Built on by (most cited first, 13 of 13)

- [It's All in the Heads: Using Attention Heads as a Baseline for Cross-Lingual Transfer in Commonsense Reasoning](https://mlsanity.com/paper/2106.12066/) (2021)
- [Natural Language Reasoning, A Survey](https://mlsanity.com/paper/2303.14725/) (2023)
- [Commonsense Knowledge Reasoning and Generation with Pre-trained Language Models: A Survey](https://mlsanity.com/paper/2201.12438/) (2022)
- [Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema](https://mlsanity.com/paper/2104.08161/) (2021)
- [The Defeat of the Winograd Schema Challenge](https://mlsanity.com/paper/2201.02387/) (2022)
- [Morph Call: Probing Morphosyntactic Content of Multilingual Transformers](https://mlsanity.com/paper/2104.12847/) (2021)
- [The Box is in the Pen: Evaluating Commonsense Reasoning in Neural Machine Translation](https://mlsanity.com/paper/2503.03308/) (2025)
- [Normal vs. Adversarial: Salience-based Analysis of Adversarial Samples for Relation Extraction](https://mlsanity.com/paper/2104.00312/) (2021)
- [Unsupervised Pronoun Resolution via Masked Noun-Phrase Prediction](https://mlsanity.com/paper/2105.12392/) (2021)
- [Enhancing Natural Language Representation with Large-Scale Out-of-Domain Commonsense](https://mlsanity.com/paper/2109.02572/) (2021)
- [Attention-based Contrastive Learning for Winograd Schemas](https://mlsanity.com/paper/2109.05108/) (2021)
- [Can Language Models perform Abductive Commonsense Reasoning?](https://mlsanity.com/paper/2207.05155/) (2022)
- [PronounFlow: A Hybrid Approach for Calibrating Pronouns in Sentences](https://mlsanity.com/paper/2308.15235/) (2023)

## Abstract (arXiv)

The recently introduced BERT model exhibits strong performance on several language understanding benchmarks. In this paper, we describe a simple re-implementation of BERT for commonsense reasoning. We show that the attentions produced by BERT can be directly utilized for tasks such as the Pronoun Disambiguation Problem and Winograd Schema Challenge. Our proposed attention-guided commonsense reasoning method is conceptually simple yet empirically powerful. Experimental analysis on multiple datasets demonstrates that our proposed system performs remarkably well on all cases while outperforming the previously reported state of the art by a margin. While results suggest that BERT seems to implicitly learn to establish complex relationships between entities, solving commonsense reasoning tasks might require more than unsupervised models learned from huge text corpora.