# Contextualized Scene Imagination for Generative Commonsense Reasoning

arXiv: https://arxiv.org/abs/2112.06318 · HTML: https://mlsanity.com/paper/2112.06318/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 40 references that are in the map.
- First posted on arXiv 12 Dec 2021.

## Builds on (most cited first, 26 of 26)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [COMET: Commonsense Transformers for Automatic Knowledge Graph Construction](https://mlsanity.com/paper/1906.05317/) (2019)
- [A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories](https://mlsanity.com/paper/1604.01696/) (2016)
- [GraphRNN: Generating Realistic Graphs with Deep Auto-regressive Models](https://mlsanity.com/paper/1802.08773/) (2018)
- [Measuring Compositional Generalization: A Comprehensive Method on Realistic Data](https://mlsanity.com/paper/1912.09713/) (2019)
- [Plan-And-Write: Towards Better Automatic Storytelling](https://mlsanity.com/paper/1811.05701/) (2018)
- [Visual Storytelling](https://mlsanity.com/paper/1604.03968/) (2016)
- [Strategies for Structuring Story Generation](https://mlsanity.com/paper/1902.01109/) (2019)
- [KG-BART: Knowledge Graph-Augmented BART for Generative Commonsense Reasoning](https://mlsanity.com/paper/2009.12677/) (2020)
- [Content Planning for Neural Story Generation with Aristotelian Rescoring](https://mlsanity.com/paper/2009.09870/) (2020)
- [FSPool: Learning Set Representations with Featurewise Sort Pooling](https://mlsanity.com/paper/1906.02795/) (2019)
- [Connecting the Dots: A Knowledgeable Path Generator for Commonsense Question Answering](https://mlsanity.com/paper/2005.00691/) (2020)
- [GPT-too: A language-model-first approach for AMR-to-text generation](https://mlsanity.com/paper/2005.09123/) (2020)
- [Sequence to Backward and Forward Sequences: A Content-Introducing Approach to Generative Short-Text Conversation](https://mlsanity.com/paper/1607.00970/) (2016)
- [RICA: Evaluating Robust Inference Capabilities Based on Commonsense Axioms](https://mlsanity.com/paper/2005.00782/) (2020)
- [An Enhanced Knowledge Injection Model for Commonsense Generation](https://mlsanity.com/paper/2012.00366/) (2020)
- [Retrieval Enhanced Model for Commonsense Generation](https://mlsanity.com/paper/2105.11174/) (2021)
- [Retrieve, Caption, Generate: Visual Grounding for Enhancing Commonsense in Text Generation Models](https://mlsanity.com/paper/2109.03892/) (2021)
- [SAPPHIRE: Approaches for Enhanced Concept-to-Text Generation](https://mlsanity.com/paper/2108.06643/) (2021)
- [Lexically-constrained Text Generation through Commonsense Knowledge Extraction and Injection](https://mlsanity.com/paper/2012.10813/) (2020)
- [KFCNet: Knowledge Filtering and Contrastive Learning Network for Generative Commonsense Reasoning](https://mlsanity.com/paper/2109.06704/) (2021)

## Built on by (most cited first, 7 of 7)

- [Visualize Before You Write: Imagination-Guided Open-Ended Text Generation](https://mlsanity.com/paper/2210.03765/) (2022)
- [DREAM: Improving Situational QA by First Elaborating the Situation](https://mlsanity.com/paper/2112.08656/) (2021)
- [Affective and Dynamic Beam Search for Story Generation](https://mlsanity.com/paper/2310.15079/) (2023)
- [SituatedGen: Incorporating Geographical and Temporal Contexts into Generative Commonsense Reasoning](https://mlsanity.com/paper/2306.12552/) (2023)
- [VER: Unifying Verbalizing Entities and Relations](https://mlsanity.com/paper/2211.11093/) (2022)
- [DECIDER: A Dual-System Rule-Controllable Decoding Framework for Language Generation](https://mlsanity.com/paper/2403.01954/) (2024)
- [Schema-Guided Culture-Aware Complex Event Simulation with Multi-Agent Role-Play](https://mlsanity.com/paper/2410.18935/) (2024)

## Abstract (arXiv)

Humans use natural language to compose common concepts from their environment into plausible, day-to-day scene descriptions. However, such generative commonsense reasoning (GCSR) skills are lacking in state-of-the-art text generation methods. Descriptive sentences about arbitrary concepts generated by neural text generation models (e.g., pre-trained text-to-text Transformers) are often grammatically fluent but may not correspond to human common sense, largely due to their lack of mechanisms to capture concept relations, to identify implicit concepts, and to perform generalizable reasoning about unseen concept compositions. In this paper, we propose an Imagine-and-Verbalize (I&V) method, which learns to imagine a relational scene knowledge graph (SKG) with relations between the input concepts, and leverage the SKG as a constraint when generating a plausible scene description. We collect and harmonize a set of knowledge resources from different domains and modalities, providing a rich auxiliary supervision signal for I&V. The experiments demonstrate the effectiveness of I&V in improving language models on both concept-to-sentence and concept-to-story generation tasks, while enabling the model to learn well from fewer task examples and generate SKGs that make common sense to human annotators.