# Attention-based Contrastive Learning for Winograd Schemas

arXiv: https://arxiv.org/abs/2109.05108 · HTML: https://mlsanity.com/paper/2109.05108/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 34 references that are in the map.
- First posted on arXiv 10 Sep 2021.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Gender Bias in Coreference Resolution](https://mlsanity.com/paper/1804.09301/) (2018)
- [Analyzing the Structure of Attention in a Transformer Language Model](https://mlsanity.com/paper/1906.04284/) (2019)
- [End-to-end Neural Coreference Resolution](https://mlsanity.com/paper/1707.07045/) (2017)
- [A Simple Method for Commonsense Reasoning](https://mlsanity.com/paper/1806.02847/) (2018)
- [The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence](https://mlsanity.com/paper/2002.06177/) (2020)
- [On Identifiability in Transformers](https://mlsanity.com/paper/1908.04211/) (2019)
- [CoDA: Contrast-enhanced and Diversity-promoting Data Augmentation for Natural Language Understanding](https://mlsanity.com/paper/2010.08670/) (2020)
- [Pre-training Is (Almost) All You Need: An Application to Commonsense Reasoning](https://mlsanity.com/paper/2004.14074/) (2020)
- [A Surprisingly Robust Trick for Winograd Schema Challenge](https://mlsanity.com/paper/1905.06290/) (2019)
- [Align, Mask and Select: A Simple Method for Incorporating Commonsense Knowledge into Language Representation Models](https://mlsanity.com/paper/1908.06725/) (2019)
- [A Review of Winograd Schema Challenge Datasets and Approaches](https://mlsanity.com/paper/2004.13831/) (2020)
- [Contrastive Self-Supervised Learning for Commonsense Reasoning](https://mlsanity.com/paper/2005.00669/) (2020)
- [How Reasonable are Common-Sense Reasoning Tasks: A Case-Study on the Winograd Schema Challenge and SWAG](https://mlsanity.com/paper/1811.01778/) (2018)
- [Attention Is (not) All You Need for Commonsense Reasoning](https://mlsanity.com/paper/1905.13497/) (2019)
- [WikiCREM: A Large Unsupervised Corpus for Coreference Resolution](https://mlsanity.com/paper/1908.08025/) (2019)
- [A Knowledge Hunting Framework for Common Sense Reasoning](https://mlsanity.com/paper/1810.01375/) (2018)
- [A Hybrid Neural Network Model for Commonsense Reasoning](https://mlsanity.com/paper/1907.11983/) (2019)
- [Solving Hard Coreference Problems](https://mlsanity.com/paper/1907.05524/) (2019)
- [Do Fine-tuned Commonsense Language Models Really Generalize?](https://mlsanity.com/paper/2011.09159/) (2020)
- [Exploring Unsupervised Pretraining and Sentence Structure Modelling for Winograd Schema Challenge](https://mlsanity.com/paper/1904.09705/) (2019)
- [A Simple Machine Learning Method for Commonsense Reasoning? A Short Commentary on Trinh & Le (2018)](https://mlsanity.com/paper/1810.00521/) (2018)

## Built on by (most cited first, 1 of 1)

- [ConVQG: Contrastive Visual Question Generation with Multimodal Guidance](https://mlsanity.com/paper/2402.12846/) (2024)

## Abstract (arXiv)

Self-supervised learning has recently attracted considerable attention in the NLP community for its ability to learn discriminative features using a contrastive objective. This paper investigates whether contrastive learning can be extended to Transfomer attention to tackling the Winograd Schema Challenge. To this end, we propose a novel self-supervised framework, leveraging a contrastive loss directly at the level of self-attention. Experimental analysis of our attention-based models on multiple datasets demonstrates superior commonsense reasoning capabilities. The proposed approach outperforms all comparable unsupervised approaches while occasionally surpassing supervised ones.