# Paragraph-level Commonsense Transformers with Recurrent Memory

arXiv: https://arxiv.org/abs/2010.01486 · HTML: https://mlsanity.com/paper/2010.01486/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Oct 2020.

## Built on by (most cited first, 23 of 23)

- [It's not Rocket Science : Interpreting Figurative Language in Narratives](https://mlsanity.com/paper/2109.00087/) (2021)
- [Entailer: Answering Questions with Faithful and Truthful Chains of Reasoning](https://mlsanity.com/paper/2210.12217/) (2022)
- [Towards Teachable Reasoning Systems: Using a Dynamic Memory of User Feedback for Continual System Improvement](https://mlsanity.com/paper/2204.13074/) (2022)
- [ValueNet: A New Dataset for Human Value Driven Dialogue System](https://mlsanity.com/paper/2112.06346/) (2021)
- [Automatic Story Generation: Challenges and Attempts](https://mlsanity.com/paper/2102.12634/) (2021)
- [Acquiring and Modelling Abstract Commonsense Knowledge via Conceptualization](https://mlsanity.com/paper/2206.01532/) (2022)
- [Mind the Gap! Injecting Commonsense Knowledge for Abstractive Dialogue Summarization](https://mlsanity.com/paper/2209.00930/) (2022)
- [GD-COMET: A Geo-Diverse Commonsense Inference Model](https://mlsanity.com/paper/2310.15383/) (2023)
- [COINS: Dynamically Generating COntextualized Inference Rules for Narrative Story Completion](https://mlsanity.com/paper/2106.02497/) (2021)
- [What happens before and after: Multi-Event Commonsense in Event Coreference Resolution](https://mlsanity.com/paper/2302.09715/) (2023)
- [Coalescing Global and Local Information for Procedural Text Understanding](https://mlsanity.com/paper/2208.12848/) (2022)
- [Commonsense Reasoning for Conversational AI: A Survey of the State of the Art](https://mlsanity.com/paper/2302.07926/) (2023)
- [Implicit Premise Generation with Discourse-aware Commonsense Knowledge Models](https://mlsanity.com/paper/2109.05358/) (2021)
- [The Power of Fragmentation: A Hierarchical Transformer Model for Structural Segmentation in Symbolic Music Generation](https://mlsanity.com/paper/2205.08579/) (2022)
- [ConvoSense: Overcoming Monotonous Commonsense Inferences for Conversational AI](https://mlsanity.com/paper/2401.15471/) (2024)
- [Automated Story Generation as Question-Answering](https://mlsanity.com/paper/2112.03808/) (2021)
- [Taxonomy of Abstractive Dialogue Summarization: Scenarios, Approaches and Future Directions](https://mlsanity.com/paper/2210.09894/) (2022)
- [A Novel LLM-based Two-stage Summarization Approach for Long Dialogues](https://mlsanity.com/paper/2410.06520/) (2024)
- [Adversarial Transformer Language Models for Contextual Commonsense Inference](https://mlsanity.com/paper/2302.05406/) (2023)
- [COMET-M: Reasoning about Multiple Events in Complex Sentences](https://mlsanity.com/paper/2305.14617/) (2023)
- [A Commonsense-Infused Language-Agnostic Learning Framework for Enhancing Prediction of Political Polarity in Multilingual News Headlines](https://mlsanity.com/paper/2212.00298/) (2022)
- [PHALM: Building a Knowledge Graph from Scratch by Prompting Humans and a Language Model](https://mlsanity.com/paper/2310.07170/) (2023)
- [Can Language Models Take A Hint? Prompting for Controllable Contextualized Commonsense Inference](https://mlsanity.com/paper/2410.02202/) (2024)

## Abstract (arXiv)

Human understanding of narrative texts requires making commonsense inferences beyond what is stated explicitly in the text. A recent model, COMET, can generate such implicit commonsense inferences along several dimensions such as pre- and post-conditions, motivations, and mental states of the participants. However, COMET was trained on commonsense inferences of short phrases, and is therefore discourse-agnostic. When presented with each sentence of a multi-sentence narrative, it might generate inferences that are inconsistent with the rest of the narrative. We present the task of discourse-aware commonsense inference. Given a sentence within a narrative, the goal is to generate commonsense inferences along predefined dimensions, while maintaining coherence with the rest of the narrative. Such large-scale paragraph-level annotation is hard to get and costly, so we use available sentence-level annotations to efficiently and automatically construct a distantly supervised corpus. Using this corpus, we train PARA-COMET, a discourse-aware model that incorporates paragraph-level information to generate coherent commonsense inferences from narratives. PARA-COMET captures both semantic knowledge pertaining to prior world knowledge, and episodic knowledge involving how current events relate to prior and future events in a narrative. Our results show that PARA-COMET outperforms the sentence-level baselines, particularly in generating inferences that are both coherent and novel.