# Contrastive Self-Supervised Learning for Commonsense Reasoning

arXiv: https://arxiv.org/abs/2005.00669 · HTML: https://mlsanity.com/paper/2005.00669/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 May 2020.

## Built on by (most cited first, 25 of 25)

- [An Empirical Study of Graph Contrastive Learning](https://mlsanity.com/paper/2109.01116/) (2021)
- [It's All in the Heads: Using Attention Heads as a Baseline for Cross-Lingual Transfer in Commonsense Reasoning](https://mlsanity.com/paper/2106.12066/) (2021)
- [TCL: Transformer-based Dynamic Graph Modelling via Contrastive Learning](https://mlsanity.com/paper/2105.07944/) (2021)
- [A Primer on Contrastive Pretraining in Language Processing: Methods, Lessons Learned and Perspectives](https://mlsanity.com/paper/2102.12982/) (2021)
- [Deep Learning Methods for Abstract Visual Reasoning: A Survey on Raven's Progressive Matrices](https://mlsanity.com/paper/2201.12382/) (2022)
- [Uncertainty-aware Contrastive Distillation for Incremental Semantic Segmentation](https://mlsanity.com/paper/2203.14098/) (2022)
- [Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema](https://mlsanity.com/paper/2104.08161/) (2021)
- [Pushing the Limits of ChatGPT on NLP Tasks](https://mlsanity.com/paper/2306.09719/) (2023)
- [Topic-Aware Contrastive Learning for Abstractive Dialogue Summarization](https://mlsanity.com/paper/2109.04994/) (2021)
- [SCD: Self-Contrastive Decorrelation for Sentence Embeddings](https://mlsanity.com/paper/2203.07847/) (2022)
- [Making Pre-trained Language Models End-to-end Few-shot Learners with Contrastive Prompt Tuning](https://mlsanity.com/paper/2204.00166/) (2022)
- [Do PLMs Know and Understand Ontological Knowledge?](https://mlsanity.com/paper/2309.05936/) (2023)
- [Improving BERT Model Using Contrastive Learning for Biomedical Relation Extraction](https://mlsanity.com/paper/2104.13913/) (2021)
- [A Sentence is Worth 128 Pseudo Tokens: A Semantic-Aware Contrastive Learning Framework for Sentence Embeddings](https://mlsanity.com/paper/2203.05877/) (2022)
- [Self-supervised Regularization for Text Classification](https://mlsanity.com/paper/2103.05231/) (2021)
- [Towards Zero-shot Commonsense Reasoning with Self-supervised Refinement of Language Models](https://mlsanity.com/paper/2109.05105/) (2021)
- [Spatial-Temporal Graph Mamba for Music-Guided Dance Video Synthesis](https://mlsanity.com/paper/2507.06689/) (2025)
- [MoNET: Tackle State Momentum via Noise-Enhanced Training for Dialogue State Tracking](https://mlsanity.com/paper/2211.05503/) (2022)
- [Constructing Contrastive samples via Summarization for Text Classification with limited annotations](https://mlsanity.com/paper/2104.05094/) (2021)
- [Unsupervised Pronoun Resolution via Masked Noun-Phrase Prediction](https://mlsanity.com/paper/2105.12392/) (2021)
- [Enhancing Natural Language Representation with Large-Scale Out-of-Domain Commonsense](https://mlsanity.com/paper/2109.02572/) (2021)
- [Attention-based Contrastive Learning for Winograd Schemas](https://mlsanity.com/paper/2109.05108/) (2021)
- [CSS: Combining Self-training and Self-supervised Learning for Few-shot Dialogue State Tracking](https://mlsanity.com/paper/2210.05146/) (2022)
- [Dictionary-Assisted Supervised Contrastive Learning](https://mlsanity.com/paper/2210.15172/) (2022)
- [Contrastive Learning for Inference in Dialogue](https://mlsanity.com/paper/2310.12467/) (2023)

## Abstract (arXiv)

We propose a self-supervised method to solve Pronoun Disambiguation and Winograd Schema Challenge problems. Our approach exploits the characteristic structure of training corpora related to so-called "trigger" words, which are responsible for flipping the answer in pronoun disambiguation. We achieve such commonsense reasoning by constructing pair-wise contrastive auxiliary predictions. To this end, we leverage a mutual exclusive loss regularized by a contrastive margin. Our architecture is based on the recently introduced transformer networks, BERT, that exhibits strong performance on many NLP benchmarks. Empirical results show that our method alleviates the limitation of current supervised approaches for commonsense reasoning. This study opens up avenues for exploiting inexpensive self-supervision to achieve performance gain in commonsense reasoning tasks.