# CIKQA: Learning Commonsense Inference with a Unified Knowledge-in-the-loop QA Paradigm

arXiv: https://arxiv.org/abs/2210.06246 · HTML: https://mlsanity.com/paper/2210.06246/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 26 of its 44 references that are in the map.
- First posted on arXiv 12 Oct 2022.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [From Recognition to Cognition: Visual Commonsense Reasoning](https://mlsanity.com/paper/1811.10830/) (2018)
- [UnifiedQA: Crossing Format Boundaries With a Single QA System](https://mlsanity.com/paper/2005.00700/) (2020)
- [KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoning](https://mlsanity.com/paper/1909.02151/) (2019)
- [Event Extraction by Answering (Almost) Natural Questions](https://mlsanity.com/paper/2004.13625/) (2020)
- [Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-trained Language Models](https://mlsanity.com/paper/2005.00683/) (2020)
- [Coreference Resolution as Query-based Span Prediction](https://mlsanity.com/paper/1911.01746/) (2019)
- [ASER: A Large-scale Eventuality Knowledge Graph](https://mlsanity.com/paper/1905.00270/) (2019)
- [GLUCOSE: GeneraLized and COntextualized Story Explanations](https://mlsanity.com/paper/2009.07758/) (2020)
- [GlossBERT: BERT for Word Sense Disambiguation with Gloss Knowledge](https://mlsanity.com/paper/1908.07245/) (2019)
- [Graph-Based Reasoning over Heterogeneous External Knowledge for Commonsense Question Answering](https://mlsanity.com/paper/1909.05311/) (2019)
- [KBQA: Learning Question Answering over QA Corpora and Knowledge Bases](https://mlsanity.com/paper/1903.02419/) (2019)
- [WinoWhy: A Deep Diagnosis of Essential Commonsense Knowledge for Answering Winograd Schema Challenge](https://mlsanity.com/paper/2005.05763/) (2020)
- [Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema](https://mlsanity.com/paper/2104.08161/) (2021)
- [Benchmarking Knowledge-Enhanced Commonsense Question Answering via Knowledge-to-Text Transformation](https://mlsanity.com/paper/2101.00760/) (2021)
- [Social Commonsense Reasoning with Multi-Head Knowledge Attention](https://mlsanity.com/paper/2010.05587/) (2020)
- [Relation Classification as Two-way Span-Prediction](https://mlsanity.com/paper/2010.04829/) (2020)
- [Do Fine-tuned Commonsense Language Models Really Generalize?](https://mlsanity.com/paper/2011.09159/) (2020)
- [A Brief Survey and Comparative Study of Recent Development of Pronoun Coreference Resolution](https://mlsanity.com/paper/2009.12721/) (2020)

## Built on by (most cited first, 2 of 2)

- [Thrust: Adaptively Propels Large Language Models with External Knowledge](https://mlsanity.com/paper/2307.10442/) (2023)
- [What Really is Commonsense Knowledge?](https://mlsanity.com/paper/2411.03964/) (2024)

## Abstract (arXiv)

Recently, the community has achieved substantial progress on many commonsense reasoning benchmarks. However, it is still unclear what is learned from the training process: the knowledge, inference capability, or both? We argue that due to the large scale of commonsense knowledge, it is infeasible to annotate a large enough training set for each task to cover all commonsense for learning. Thus we should separate the commonsense knowledge acquisition and inference over commonsense knowledge as two separate tasks. In this work, we focus on investigating models' commonsense inference capabilities from two perspectives: (1) Whether models can know if the knowledge they have is enough to solve the task; (2) Whether models can develop commonsense inference capabilities that generalize across commonsense tasks. We first align commonsense tasks with relevant knowledge from commonsense knowledge bases and ask humans to annotate whether the knowledge is enough or not. Then, we convert different commonsense tasks into a unified question answering format to evaluate models' generalization capabilities. We name the benchmark as Commonsense Inference with Knowledge-in-the-loop Question Answering (CIKQA).