# A Knowledge-Enhanced Pretraining Model for Commonsense Story Generation

arXiv: https://arxiv.org/abs/2001.05139 · HTML: https://mlsanity.com/paper/2001.05139/

## Key facts

- Cited by 116 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jan 2020.

## Built on by (most cited first, 30 of 116)

- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [Pre-Trained Models: Past, Present and Future](https://mlsanity.com/paper/2106.07139/) (2021)
- [Co-Writing Screenplays and Theatre Scripts with Language Models: An Evaluation by Industry Professionals](https://mlsanity.com/paper/2209.14958/) (2022)
- [Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification](https://mlsanity.com/paper/2108.02035/) (2021)
- [Pretrained Language Models for Text Generation: A Survey](https://mlsanity.com/paper/2201.05273/) (2022)
- [Locally Typical Sampling](https://mlsanity.com/paper/2202.00666/) (2022)
- [A Confederacy of Models: a Comprehensive Evaluation of LLMs on Creative Writing](https://mlsanity.com/paper/2310.08433/) (2023)
- [The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation](https://mlsanity.com/paper/2109.06835/) (2021)
- [Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey](https://mlsanity.com/paper/2210.07700/) (2022)
- [A Survey of Knowledge Enhanced Pre-trained Language Models](https://mlsanity.com/paper/2211.05994/) (2022)
- [Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generation](https://mlsanity.com/paper/2208.11646/) (2022)
- [OpenMEVA: A Benchmark for Evaluating Open-ended Story Generation Metrics](https://mlsanity.com/paper/2105.08920/) (2021)
- [The Next Chapter: A Study of Large Language Models in Storytelling](https://mlsanity.com/paper/2301.09790/) (2023)
- [Give Us the Facts: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modeling](https://mlsanity.com/paper/2306.11489/) (2023)
- [Understanding the Capabilities and Limitations of Large Language Models for Cultural Commonsense](https://mlsanity.com/paper/2405.04655/) (2024)
- [Long Text Generation by Modeling Sentence-Level and Discourse-Level Coherence](https://mlsanity.com/paper/2105.08963/) (2021)
- [ClarET: Pre-training a Correlation-Aware Context-To-Event Transformer for Event-Centric Generation and Classification](https://mlsanity.com/paper/2203.02225/) (2022)
- [Generate, Annotate, and Learn: NLP with Synthetic Text](https://mlsanity.com/paper/2106.06168/) (2021)
- [Dict-BERT: Enhancing Language Model Pre-training with Dictionary](https://mlsanity.com/paper/2110.06490/) (2021)
- [CICERO: A Dataset for Contextualized Commonsense Inference in Dialogues](https://mlsanity.com/paper/2203.13926/) (2022)
- [Knowledge Enhanced Pretrained Language Models: A Compreshensive Survey](https://mlsanity.com/paper/2110.08455/) (2021)
- [Uncovering Implicit Gender Bias in Narratives through Commonsense Inference](https://mlsanity.com/paper/2109.06437/) (2021)
- [Do Language Models Enjoy Their Own Stories? Prompting Large Language Models for Automatic Story Evaluation](https://mlsanity.com/paper/2405.13769/) (2024)
- [Relational World Knowledge Representation in Contextual Language Models: A Review](https://mlsanity.com/paper/2104.05837/) (2021)
- [Visual Writing Prompts: Character-Grounded Story Generation with Curated Image Sequences](https://mlsanity.com/paper/2301.08571/) (2023)
- [Lawyers are Dishonest? Quantifying Representational Harms in Commonsense Knowledge Resources](https://mlsanity.com/paper/2103.11320/) (2021)
- [Visualize Before You Write: Imagination-Guided Open-Ended Text Generation](https://mlsanity.com/paper/2210.03765/) (2022)
- [Inferring the Reader: Guiding Automated Story Generation with Commonsense Reasoning](https://mlsanity.com/paper/2105.01311/) (2021)
- [Automatic Story Generation: Challenges and Attempts](https://mlsanity.com/paper/2102.12634/) (2021)
- [DISCOS: Bridging the Gap between Discourse Knowledge and Commonsense Knowledge](https://mlsanity.com/paper/2101.00154/) (2021)

## Abstract (arXiv)

Story generation, namely generating a reasonable story from a leading context, is an important but challenging task. In spite of the success in modeling fluency and local coherence, existing neural language generation models (e.g., GPT-2) still suffer from repetition, logic conflicts, and lack of long-range coherence in generated stories. We conjecture that this is because of the difficulty of associating relevant commonsense knowledge, understanding the causal relationships, and planning entities and events with proper temporal order. In this paper, we devise a knowledge-enhanced pretraining model for commonsense story generation. We propose to utilize commonsense knowledge from external knowledge bases to generate reasonable stories. To further capture the causal and temporal dependencies between the sentences in a reasonable story, we employ multi-task learning which combines a discriminative objective to distinguish true and fake stories during fine-tuning. Automatic and manual evaluation shows that our model can generate more reasonable stories than state-of-the-art baselines, particularly in terms of logic and global coherence.