# Exploring Strategies for Generalizable Commonsense Reasoning with Pre-trained Models

arXiv: https://arxiv.org/abs/2109.02837 · HTML: https://mlsanity.com/paper/2109.02837/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 40 references that are in the map.
- First posted on arXiv 7 Sep 2021.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://mlsanity.com/paper/1707.07328/) (2017)
- [What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models](https://mlsanity.com/paper/1907.13528/) (2019)
- [Abductive Commonsense Reasoning](https://mlsanity.com/paper/1908.05739/) (2019)
- [Unsupervised Commonsense Question Answering with Self-Talk](https://mlsanity.com/paper/2004.05483/) (2020)
- [How Much Reading Does Reading Comprehension Require? A Critical Investigation of Popular Benchmarks](https://mlsanity.com/paper/1808.04926/) (2018)
- [Look at the First Sentence: Position Bias in Question Answering](https://mlsanity.com/paper/2004.14602/) (2020)
- [We Need to Talk About Random Splits](https://mlsanity.com/paper/2005.00636/) (2020)
- [ProtoQA: A Question Answering Dataset for Prototypical Common-Sense Reasoning](https://mlsanity.com/paper/2005.00771/) (2020)
- [What do Models Learn from Question Answering Datasets?](https://mlsanity.com/paper/2004.03490/) (2020)
- [Towards Generalizable Neuro-Symbolic Systems for Commonsense Question Answering](https://mlsanity.com/paper/1910.14087/) (2019)
- [RICA: Evaluating Robust Inference Capabilities Based on Commonsense Axioms](https://mlsanity.com/paper/2005.00782/) (2020)
- [What Does My QA Model Know? Devising Controlled Probes using Expert Knowledge](https://mlsanity.com/paper/1912.13337/) (2019)
- [Undersensitivity in Neural Reading Comprehension](https://mlsanity.com/paper/2003.04808/) (2020)
- [Identify, Align, and Integrate: Matching Knowledge Graphs to Commonsense Reasoning Tasks](https://mlsanity.com/paper/2104.10193/) (2021)
- [What do we expect from Multiple-choice QA Systems?](https://mlsanity.com/paper/2011.10647/) (2020)

## Built on by (most cited first, 11 of 11)

- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [ChatGPT is a Knowledgeable but Inexperienced Solver: An Investigation of Commonsense Problem in Large Language Models](https://mlsanity.com/paper/2303.16421/) (2023)
- [Visualize Before You Write: Imagination-Guided Open-Ended Text Generation](https://mlsanity.com/paper/2210.03765/) (2022)
- [BRAINTEASER: Lateral Thinking Puzzles for Large Language Models](https://mlsanity.com/paper/2310.05057/) (2023)
- [A Study of Situational Reasoning for Traffic Understanding](https://mlsanity.com/paper/2306.02520/) (2023)
- [Robust and Explainable Identification of Logical Fallacies in Natural Language Arguments](https://mlsanity.com/paper/2212.07425/) (2022)
- [Coalescing Global and Local Information for Procedural Text Understanding](https://mlsanity.com/paper/2208.12848/) (2022)
- [An Empirical Investigation of Commonsense Self-Supervision with Knowledge Graphs](https://mlsanity.com/paper/2205.10661/) (2022)
- [Generalizable Neuro-symbolic Systems for Commonsense Question Answering](https://mlsanity.com/paper/2201.06230/) (2022)
- [Utilizing Background Knowledge for Robust Reasoning over Traffic Situations](https://mlsanity.com/paper/2212.07798/) (2022)
- [KEPR: Knowledge Enhancement and Plausibility Ranking for Generative Commonsense Question Answering](https://mlsanity.com/paper/2305.08347/) (2023)

## Abstract (arXiv)

Commonsense reasoning benchmarks have been largely solved by fine-tuning language models. The downside is that fine-tuning may cause models to overfit to task-specific data and thereby forget their knowledge gained during pre-training. Recent works only propose lightweight model updates as models may already possess useful knowledge from past experience, but a challenge remains in understanding what parts and to what extent models should be refined for a given task. In this paper, we investigate what models learn from commonsense reasoning datasets. We measure the impact of three different adaptation methods on the generalization and accuracy of models. Our experiments with two models show that fine-tuning performs best, by learning both the content and the structure of the task, but suffers from overfitting and limited generalization to novel answers. We observe that alternative adaptation methods like prefix-tuning have comparable accuracy, but generalize better to unseen answers and are more robust to adversarial splits.