# Plot and Rework: Modeling Storylines for Visual Storytelling

arXiv: https://arxiv.org/abs/2105.06950 · HTML: https://mlsanity.com/paper/2105.06950/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 41 references that are in the map.
- First posted on arXiv 14 May 2021.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories](https://mlsanity.com/paper/1604.01696/) (2016)
- [Image Generation from Scene Graphs](https://mlsanity.com/paper/1804.01622/) (2018)
- [Plan-And-Write: Towards Better Automatic Storytelling](https://mlsanity.com/paper/1811.05701/) (2018)
- [Visual Storytelling](https://mlsanity.com/paper/1604.03968/) (2016)
- [Strategies for Structuring Story Generation](https://mlsanity.com/paper/1902.01109/) (2019)
- [Towards Diverse and Natural Image Descriptions via a Conditional GAN](https://mlsanity.com/paper/1703.06029/) (2017)
- [No Metrics Are Perfect: Adversarial Reward Learning for Visual Storytelling](https://mlsanity.com/paper/1804.09160/) (2018)
- [UHop: An Unrestricted-Hop Relation Extraction Framework for Knowledge-Based Question Answering](https://mlsanity.com/paper/1904.01246/) (2019)
- [Frame-Semantic Parsing with Softmax-Margin Segmental RNNs and a Syntactic Scaffold](https://mlsanity.com/paper/1706.09528/) (2017)
- [What Makes A Good Story? Designing Composite Rewards for Visual Storytelling](https://mlsanity.com/paper/1909.05316/) (2019)
- [Knowledge-Enriched Visual Storytelling](https://mlsanity.com/paper/1912.01496/) (2019)
- [GLAC Net: GLocal Attention Cascading Networks for Multi-image Cued Story Generation](https://mlsanity.com/paper/1805.10973/) (2018)
- [Hierarchically Structured Reinforcement Learning for Topically Coherent Visual Story Generation](https://mlsanity.com/paper/1805.08191/) (2018)
- [Contextualize, Show and Tell: A Neural Visual Storyteller](https://mlsanity.com/paper/1806.00738/) (2018)
- [Hide-and-Tell: Learning to Bridge Photo Streams for Visual Storytelling](https://mlsanity.com/paper/2002.00774/) (2020)
- [Visual Story Post-Editing](https://mlsanity.com/paper/1906.01764/) (2019)
- [Topic Adaptation and Prototype Encoding for Few-Shot Visual Storytelling](https://mlsanity.com/paper/2008.04504/) (2020)
- [The Steep Road to Happily Ever After: An Analysis of Current Visual Storytelling Models](https://mlsanity.com/paper/1904.03366/) (2019)
- [Multimodal Storytelling via Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1712.01455/) (2017)
- [Using Inter-Sentence Diverse Beam Search to Reduce Redundancy in Visual Storytelling](https://mlsanity.com/paper/1805.11867/) (2018)
- [Keep it Consistent: Topic-Aware Storytelling from an Image Stream via Iterative Multi-agent Communication](https://mlsanity.com/paper/1911.04192/) (2019)
- [Knowledge Base Relation Detection via Multi-View Matching](https://mlsanity.com/paper/1803.00612/) (2018)

## Built on by (most cited first, 14 of 14)

- [VIST-GPT: Ushering in the Era of Visual Storytelling with LLMs?](https://mlsanity.com/paper/2504.19267/) (2025)
- [Detecting and Grounding Important Characters in Visual Stories](https://mlsanity.com/paper/2303.17647/) (2023)
- [Generating Coherent Sequences of Visual Illustrations for Real-World Manual Tasks](https://mlsanity.com/paper/2405.10122/) (2024)
- [Visual Storytelling with Question-Answer Plans](https://mlsanity.com/paper/2310.05295/) (2023)
- [SCO-VIST: Social Interaction Commonsense Knowledge-based Visual Storytelling](https://mlsanity.com/paper/2402.00319/) (2024)
- [Not (yet) the whole story: Evaluating Visual Storytelling Requires More than Measuring Coherence, Grounding, and Repetition](https://mlsanity.com/paper/2407.04559/) (2024)
- [Story Generation from Visual Inputs: Techniques, Related Tasks, and Challenges](https://mlsanity.com/paper/2406.02748/) (2024)
- [Only Send What You Need: Learning to Communicate Efficiently in Federated Multilingual Machine Translation](https://mlsanity.com/paper/2401.07456/) (2024)
- [Generating Visual Stories with Grounded and Coreferent Characters](https://mlsanity.com/paper/2409.13555/) (2024)
- [MDSF: Context-Aware Multi-Dimensional Data Storytelling Framework based on Large language Model](https://mlsanity.com/paper/2501.01014/) (2025)
- [TARN-VIST: Topic Aware Reinforcement Network for Visual Storytelling](https://mlsanity.com/paper/2403.11550/) (2024)
- [A Pattern to Align Them All: Integrating Different Modalities to Define Multi-Modal Entities](https://mlsanity.com/paper/2410.13803/) (2024)
- [From Image Captioning to Visual Storytelling](https://mlsanity.com/paper/2508.14045/) (2025)
- [StoryMovie: A Dataset for Semantic Alignment of Visual Stories with Movie Scripts and Subtitles](https://mlsanity.com/paper/2602.21829/) (2026)

## Abstract (arXiv)

Writing a coherent and engaging story is not easy. Creative writers use their knowledge and worldview to put disjointed elements together to form a coherent storyline, and work and rework iteratively toward perfection. Automated visual storytelling (VIST) models, however, make poor use of external knowledge and iterative generation when attempting to create stories. This paper introduces PR-VIST, a framework that represents the input image sequence as a story graph in which it finds the best path to form a storyline. PR-VIST then takes this path and learns to generate the final story via an iterative training process. This framework produces stories that are superior in terms of diversity, coherence, and humanness, per both automatic and human evaluations. An ablation study shows that both plotting and reworking contribute to the model's superiority.