# Exploring Story Generation with Multi-task Objectives in Variational Autoencoders

arXiv: https://arxiv.org/abs/2111.08133 · HTML: https://mlsanity.com/paper/2111.08133/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 35 references that are in the map.
- First posted on arXiv 15 Nov 2021.

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [An Introduction to Variational Autoencoders](https://mlsanity.com/paper/1906.02691/) (2019)
- [Texygen: A Benchmarking Platform for Text Generation Models](https://mlsanity.com/paper/1802.01886/) (2018)
- [Importance Weighted Autoencoders](https://mlsanity.com/paper/1509.00519/) (2015)
- [Understanding disentangling in β-VAE](https://mlsanity.com/paper/1804.03599/) (2018)
- [Plan-And-Write: Towards Better Automatic Storytelling](https://mlsanity.com/paper/1811.05701/) (2018)
- [Unifying Human and Statistical Evaluation for Natural Language Generation](https://mlsanity.com/paper/1904.02792/) (2019)
- [Lagging Inference Networks and Posterior Collapse in Variational Autoencoders](https://mlsanity.com/paper/1901.05534/) (2019)
- [A Knowledge-Enhanced Pretraining Model for Commonsense Story Generation](https://mlsanity.com/paper/2001.05139/) (2020)
- [Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space](https://mlsanity.com/paper/2004.04092/) (2020)
- [Language GANs Falling Short](https://mlsanity.com/paper/1811.02549/) (2018)
- [MEGATRON-CNTRL: Controllable Story Generation with External Knowledge Using Large-Scale Language Models](https://mlsanity.com/paper/2010.00840/) (2020)
- [Jointly Measuring Diversity and Quality in Text Generation Models](https://mlsanity.com/paper/1904.03971/) (2019)
- [UNION: An Unreferenced Metric for Evaluating Open-ended Story Generation](https://mlsanity.com/paper/2009.07602/) (2020)
- [Long Text Generation by Modeling Sentence-Level and Discourse-Level Coherence](https://mlsanity.com/paper/2105.08963/) (2021)
- [A Surprisingly Effective Fix for Deep Latent Variable Modeling of Text](https://mlsanity.com/paper/1909.00868/) (2019)
- [Automatic Story Generation: Challenges and Attempts](https://mlsanity.com/paper/2102.12634/) (2021)
- [Narrative Text Generation with a Latent Discrete Plan](https://mlsanity.com/paper/2010.03272/) (2020)
- [An Automatic Approach for Document-level Topic Model Evaluation](https://mlsanity.com/paper/1706.05140/) (2017)
- [Do sequence-to-sequence VAEs learn global features of sentences?](https://mlsanity.com/paper/2004.07683/) (2020)

## Built on by (most cited first, 4 of 4)

- [The Next Chapter: A Study of Large Language Models in Storytelling](https://mlsanity.com/paper/2301.09790/) (2023)
- [Towards Inter-character Relationship-driven Story Generation](https://mlsanity.com/paper/2211.00676/) (2022)
- [DeltaScore: Fine-Grained Story Evaluation with Perturbations](https://mlsanity.com/paper/2303.08991/) (2023)
- [The Free Transformer](https://mlsanity.com/paper/2510.17558/) (2025)

## Abstract (arXiv)

GPT-2 has been frequently adapted in story generation models as it provides powerful generative capability. However, it still fails to generate consistent stories and lacks diversity. Current story generation models leverage additional information such as plots or commonsense into GPT-2 to guide the generation process. These approaches focus on improving generation quality of stories while our work look at both quality and diversity. We explore combining BERT and GPT-2 to build a variational autoencoder (VAE), and extend it by adding additional objectives to learn global features such as story topic and discourse relations. Our evaluations show our enhanced VAE can provide better quality and diversity trade off, generate less repetitive story content and learn a more informative latent variable.