mlsanityData to 7 Oct 2026

Towards Zero-shot Commonsense Reasoning with Self-supervised Refinement of Language Models

arXiv 2109.05105 ↗First posted 10 Sep 2021cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Can we get existing language models and refine them for zero-shot commonsense reasoning? This paper presents an initial study exploring the feasibility of zero-shot commonsense reasoning for the Winograd Schema Challenge by formulating the task as self-supervised refinement of a pre-trained language model. In contrast to previous studies that rely on fine-tuning annotated datasets, we seek to boost conceptualization via loss landscape refinement. To this end, we propose a novel self-supervised learning approach that refines the language model utilizing a set of linguistic perturbations of similar concept relationships. Empirical analysis of our conceptually simple framework demonstrates the viability of zero-shot commonsense reasoning on multiple benchmarks.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 20 of its 30 references that are in the map. First posted on arXiv 10 Sep 2021.

Builds on

20 of 30 references.

References · 20

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  7. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 657
  8. 2018Gender Bias in Coreference Resolutioncited by 438
  9. 2019Unmasking Clever Hans Predictors and Assessing What Machines Really Learncited by 359
  10. 2020The Next Decade in AI: Four Steps Towards Robust Artificial Intelligencecited by 194
  11. 2018Robustness via curvature regularization, and vice versacited by 174
  12. 2019Adversarial Robustness through Local Linearizationcited by 146
8 more
  1. 2019A Surprisingly Robust Trick for Winograd Schema Challengecited by 30
  2. 2020A Review of Winograd Schema Challenge Datasets and Approachescited by 27
  3. 2020Contrastive Self-Supervised Learning for Commonsense Reasoningcited by 25
  4. 2020The Sensitivity of Language Models and Humans to Winograd Schema Perturbationscited by 21
  5. 2018How Reasonable are Common-Sense Reasoning Tasks: A Case-Study on the Winograd Schema Challenge and SWAGcited by 18
  6. 2020Do Fine-tuned Commonsense Language Models Really Generalize?cited by 8
  7. 2020Precise Task Formalization Matters in Winograd Schema Evaluationscited by 7
  8. 2018A Simple Machine Learning Method for Commonsense Reasoning? A Short Commentary on Trinh & Le (2018)cited by 2

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 3.

4 papers.

  1. 2023ChatGPT is a Knowledgeable but Inexperienced Solver: An Investigation of Commonsense Problem in Large Language Modelscited by 63
  2. 2022minicons: Enabling Flexible Behavioral and Representational Analyses of Transformer Language Modelscited by 58
  3. 2023Commonsense Knowledge Transfer for Pre-trained Language Modelscited by 5
  4. 2023CASE: Commonsense-Augmented Score with an Expanded Answer Spacecited by 0