# CoCoLM: COmplex COmmonsense Enhanced Language Model with Discourse Relations

arXiv: https://arxiv.org/abs/2012.15643 · HTML: https://mlsanity.com/paper/2012.15643/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Dec 2020.

## Built on by (most cited first, 15 of 15)

- [ClarET: Pre-training a Correlation-Aware Context-To-Event Transformer for Event-Centric Generation and Classification](https://mlsanity.com/paper/2203.02225/) (2022)
- [EventBERT: A Pre-Trained Model for Event Correlation Reasoning](https://mlsanity.com/paper/2110.06533/) (2021)
- [Knowledge Enhanced Pretrained Language Models: A Compreshensive Survey](https://mlsanity.com/paper/2110.08455/) (2021)
- [ASER: Towards Large-scale Commonsense Knowledge Acquisition via Higher-order Selectional Preference over Eventualities](https://mlsanity.com/paper/2104.02137/) (2021)
- [CAT: A Contextualized Conceptualization and Instantiation Framework for Commonsense Reasoning](https://mlsanity.com/paper/2305.04808/) (2023)
- [An Overview Of Temporal Commonsense Reasoning and Acquisition](https://mlsanity.com/paper/2308.00002/) (2023)
- [EventGround: Narrative Reasoning by Grounding to Eventuality-centric Knowledge Graphs](https://mlsanity.com/paper/2404.00209/) (2024)
- [A Survey of Implicit Discourse Relation Recognition](https://mlsanity.com/paper/2203.02982/) (2022)
- [An Interpretable Ensemble of Graph and Language Models for Improving Search Relevance in E-Commerce](https://mlsanity.com/paper/2403.00923/) (2024)
- [Event Transition Planning for Open-ended Text Generation](https://mlsanity.com/paper/2204.09453/) (2022)
- [A Survey on Knowledge-Enhanced Pre-trained Language Models](https://mlsanity.com/paper/2212.13428/) (2022)
- [Temporal Validity Change Prediction](https://mlsanity.com/paper/2401.00779/) (2024)
- [EVIT: Event-Oriented Instruction Tuning for Event Reasoning](https://mlsanity.com/paper/2404.11978/) (2024)
- [What Really is Commonsense Knowledge?](https://mlsanity.com/paper/2411.03964/) (2024)
- [Meta-Semantics Augmented Few-Shot Relational Learning](https://mlsanity.com/paper/2505.05684/) (2025)

## Abstract (arXiv)

Large-scale pre-trained language models have demonstrated strong knowledge representation ability. However, recent studies suggest that even though these giant models contains rich simple commonsense knowledge (e.g., bird can fly and fish can swim.), they often struggle with the complex commonsense knowledge that involves multiple eventualities (verb-centric phrases, e.g., identifying the relationship between ``Jim yells at Bob'' and ``Bob is upset'').To address this problem, in this paper, we propose to help pre-trained language models better incorporate complex commonsense knowledge. Different from existing fine-tuning approaches, we do not focus on a specific task and propose a general language model named CoCoLM. Through the careful training over a large-scale eventuality knowledge graphs ASER, we successfully teach pre-trained language models (i.e., BERT and RoBERTa) rich complex commonsense knowledge among eventualities. Experiments on multiple downstream commonsense tasks that requires the correct understanding of eventualities demonstrate the effectiveness of CoCoLM.