# CORECODE: A Common Sense Annotated Dialogue Dataset with Benchmark Tasks for Chinese Large Language Models

arXiv: https://arxiv.org/abs/2312.12853 · HTML: https://mlsanity.com/paper/2312.12853/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 35 of its 49 references that are in the map.
- First posted on arXiv 20 Dec 2023.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 30 of 35)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://mlsanity.com/paper/2103.10360/) (2021)
- [GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](https://mlsanity.com/paper/2112.06905/) (2021)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca](https://mlsanity.com/paper/2304.08177/) (2023)
- [Symbolic Knowledge Distillation: from General Language Models to Commonsense Models](https://mlsanity.com/paper/2110.07178/) (2021)
- [COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphs](https://mlsanity.com/paper/2010.05953/) (2020)
- [QASC: A Dataset for Question Answering via Sentence Composition](https://mlsanity.com/paper/1910.11473/) (2019)
- [KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoning](https://mlsanity.com/paper/1909.02151/) (2019)
- [Generated Knowledge Prompting for Commonsense Reasoning](https://mlsanity.com/paper/2110.08387/) (2021)
- [Evaluating Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2310.19736/) (2023)
- ["Going on a vacation" takes longer than "Going for a walk": A Study of Temporal Commonsense Understanding](https://mlsanity.com/paper/1909.03065/) (2019)
- [Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-trained Language Models](https://mlsanity.com/paper/2005.00683/) (2020)
- [Long Time No See! Open-Domain Conversation with Long-Term Persona Memory](https://mlsanity.com/paper/2203.05797/) (2022)
- [KG-BART: Knowledge Graph-Augmented BART for Generative Commonsense Reasoning](https://mlsanity.com/paper/2009.12677/) (2020)
- [Evaluating Commonsense in Pre-trained Language Models](https://mlsanity.com/paper/1911.11931/) (2019)
- [Graph-Based Reasoning over Heterogeneous External Knowledge for Commonsense Question Answering](https://mlsanity.com/paper/1909.05311/) (2019)
- [CrossWOZ: A Large-Scale Chinese Cross-Domain Task-Oriented Dialogue Dataset](https://mlsanity.com/paper/2002.11893/) (2020)
- [Recent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and Approaches](https://mlsanity.com/paper/1904.01172/) (2019)
- [TIMEDIAL: Temporal Commonsense Reasoning in Dialog](https://mlsanity.com/paper/2106.04571/) (2021)
- [Commonsense for Generative Multi-Hop Question Answering Tasks](https://mlsanity.com/paper/1809.06309/) (2018)
- [ClarET: Pre-training a Correlation-Aware Context-To-Event Transformer for Event-Centric Generation and Classification](https://mlsanity.com/paper/2203.02225/) (2022)
- [Connecting the Dots: A Knowledgeable Path Generator for Commonsense Question Answering](https://mlsanity.com/paper/2005.00691/) (2020)
- [RiSAWOZ: A Large-Scale Multi-Domain Wizard-of-Oz Dataset with Rich Semantic Annotations for Task-Oriented Dialogue Modeling](https://mlsanity.com/paper/2010.08738/) (2020)

## Built on by (most cited first, 3 of 3)

- [IRCAN: Mitigating Knowledge Conflicts in LLM Generation via Identifying and Reweighting Context-Aware Neurons](https://mlsanity.com/paper/2406.18406/) (2024)
- [Benchmarking Chinese Commonsense Reasoning of LLMs: From Chinese-Specifics to Reasoning-Memorization Correlations](https://mlsanity.com/paper/2403.14112/) (2024)
- [The Illusion of Equivalency: Statistical Characterization of Quantization Effects in LLMs](https://mlsanity.com/paper/2607.08734/) (2026)

## Abstract (arXiv)

As an indispensable ingredient of intelligence, commonsense reasoning is crucial for large language models (LLMs) in real-world scenarios. In this paper, we propose CORECODE, a dataset that contains abundant commonsense knowledge manually annotated on dyadic dialogues, to evaluate the commonsense reasoning and commonsense conflict detection capabilities of Chinese LLMs. We categorize commonsense knowledge in everyday conversations into three dimensions: entity, event, and social interaction. For easy and consistent annotation, we standardize the form of commonsense knowledge annotation in open-domain dialogues as "domain: slot = value". A total of 9 domains and 37 slots are defined to capture diverse commonsense knowledge. With these pre-defined domains and slots, we collect 76,787 commonsense knowledge annotations from 19,700 dialogues through crowdsourcing. To evaluate and enhance the commonsense reasoning capability for LLMs on the curated dataset, we establish a series of dialogue-level reasoning and detection tasks, including commonsense knowledge filling, commonsense knowledge generation, commonsense conflict phrase detection, domain identification, slot identification, and event causal inference. A wide variety of existing open-source Chinese LLMs are evaluated with these tasks on our dataset. Experimental results demonstrate that these models are not competent to predict CORECODE's plentiful reasoning content, and even ChatGPT could only achieve 0.275 and 0.084 accuracy on the domain identification and slot identification tasks under the zero-shot setting. We release the data and codes of CORECODE at https://github.com/danshi777/CORECODE to promote commonsense reasoning evaluation and study of LLMs in the context of daily conversations.