# Visual Writing Prompts: Character-Grounded Story Generation with Curated Image Sequences

arXiv: https://arxiv.org/abs/2301.08571 · HTML: https://mlsanity.com/paper/2301.08571/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 57 references that are in the map.
- First posted on arXiv 20 Jan 2023.

## Builds on (most cited first, 26 of 26)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [Cascade R-CNN: High Quality Object Detection and Instance Segmentation](https://mlsanity.com/paper/1906.09756/) (2019)
- [A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories](https://mlsanity.com/paper/1604.01696/) (2016)
- [The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics](https://mlsanity.com/paper/2102.01672/) (2021)
- [Plan-And-Write: Towards Better Automatic Storytelling](https://mlsanity.com/paper/1811.05701/) (2018)
- [MovieNet: A Holistic Dataset for Movie Understanding](https://mlsanity.com/paper/2007.10937/) (2020)
- [Visual Storytelling](https://mlsanity.com/paper/1604.03968/) (2016)
- [Simple BERT Models for Relation Extraction and Semantic Role Labeling](https://mlsanity.com/paper/1904.05255/) (2019)
- [Higher-order Coreference Resolution with Coarse-to-fine Inference](https://mlsanity.com/paper/1804.05392/) (2018)
- [Strategies for Structuring Story Generation](https://mlsanity.com/paper/1902.01109/) (2019)
- [A Knowledge-Enhanced Pretraining Model for Commonsense Story Generation](https://mlsanity.com/paper/2001.05139/) (2020)
- [PlotMachines: Outline-Conditioned Generation with Dynamic Plot State Tracking](https://mlsanity.com/paper/2004.14967/) (2020)
- [Content Planning for Neural Story Generation with Aristotelian Rescoring](https://mlsanity.com/paper/2009.09870/) (2020)
- [Counterfactual Story Reasoning and Generation](https://mlsanity.com/paper/1909.04076/) (2019)
- [Semi-supervised sequence tagging with bidirectional language models](https://mlsanity.com/paper/1705.00108/) (2017)
- [MEGATRON-CNTRL: Controllable Story Generation with External Knowledge Using Large-Scale Language Models](https://mlsanity.com/paper/2010.00840/) (2020)
- [Event Representations for Automated Story Generation with Deep Neural Nets](https://mlsanity.com/paper/1706.01331/) (2017)
- [Modeling Naive Psychology of Characters in Simple Commonsense Stories](https://mlsanity.com/paper/1805.06533/) (2018)
- [UNION: An Unreferenced Metric for Evaluating Open-ended Story Generation](https://mlsanity.com/paper/2009.07602/) (2020)
- [STORIUM: A Dataset and Evaluation Platform for Machine-in-the-Loop Story Generation](https://mlsanity.com/paper/2010.01717/) (2020)
- [A Graph-Based Framework to Bridge Movies and Synopses](https://mlsanity.com/paper/1910.11009/) (2019)
- [Knowledge-Enriched Visual Storytelling](https://mlsanity.com/paper/1912.01496/) (2019)

## Built on by (most cited first, 30 of 31)

- [A Design Space for Intelligent and Interactive Writing Assistants](https://mlsanity.com/paper/2403.14117/) (2024)
- [Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Art](https://mlsanity.com/paper/2406.03930/) (2024)
- [Evaluating Creative Short Story Generation in Humans and Large Language Models](https://mlsanity.com/paper/2411.02316/) (2024)
- [What Makes a Good Story and How Can We Measure It? A Comprehensive Survey of Story Evaluation](https://mlsanity.com/paper/2408.14622/) (2024)
- [From MOOC to MAIC: Reshaping Online Teaching and Learning through LLM-driven Agents](https://mlsanity.com/paper/2409.03512/) (2024)
- [Event Causality Is Key to Computational Story Understanding](https://mlsanity.com/paper/2311.09648/) (2023)
- [Tails Tell Tales: Chapter-Wide Manga Transcriptions with Character Names](https://mlsanity.com/paper/2408.00298/) (2024)
- [GROOViST: A Metric for Grounding Objects in Visual Storytelling](https://mlsanity.com/paper/2310.17770/) (2023)
- [Creativity in AI: Progresses and Challenges](https://mlsanity.com/paper/2410.17218/) (2024)
- [ScriptViz: A Visualization Tool to Aid Scriptwriting based on a Large Movie Database](https://mlsanity.com/paper/2410.03224/) (2024)
- [Not (yet) the whole story: Evaluating Visual Storytelling Requires More than Measuring Coherence, Grounding, and Repetition](https://mlsanity.com/paper/2407.04559/) (2024)
- [LongWriter-V: Enabling Ultra-Long and High-Fidelity Generation in Vision-Language Models](https://mlsanity.com/paper/2502.14834/) (2025)
- [HL Dataset: Visually-grounded Description of Scenes, Actions and Rationales](https://mlsanity.com/paper/2302.12189/) (2023)
- [Semi-supervised multimodal coreference resolution in image narrations](https://mlsanity.com/paper/2310.13619/) (2023)
- [VinaBench: Benchmark for Faithful and Consistent Visual Narratives](https://mlsanity.com/paper/2503.20871/) (2025)
- [Burn After Reading: Do Multimodal Large Language Models Truly Capture Order of Events in Image Sequences?](https://mlsanity.com/paper/2506.10415/) (2025)
- [Train & Constrain: Phonologically Informed Tongue-Twister Generation from Topics and Paraphrases](https://mlsanity.com/paper/2403.13901/) (2024)
- [Story Generation from Visual Inputs: Techniques, Related Tasks, and Challenges](https://mlsanity.com/paper/2406.02748/) (2024)
- [Narrative Studio: Visual narrative exploration using LLMs and Monte Carlo Tree Search](https://mlsanity.com/paper/2504.02426/) (2025)
- [StoryReasoning Dataset: Using Chain-of-Thought for Scene Understanding and Grounded Story Generation](https://mlsanity.com/paper/2505.10292/) (2025)
- [Improving Visual Storytelling with Multimodal Large Language Models](https://mlsanity.com/paper/2407.02586/) (2024)
- [Context-aware Visual Storytelling with Visual Prefix Tuning and Contrastive Learning](https://mlsanity.com/paper/2408.06259/) (2024)
- [Generating Visual Stories with Grounded and Coreferent Characters](https://mlsanity.com/paper/2409.13555/) (2024)
- [Mitigating Behavioral Hallucination in Multimodal Large Language Models for Sequential Images](https://mlsanity.com/paper/2506.07184/) (2025)
- [LLMs Behind the Scenes: Enabling Narrative Scene Illustration](https://mlsanity.com/paper/2509.22940/) (2025)
- [ReCap: Lightweight Referential Grounding for Coherent Story Visualization](https://mlsanity.com/paper/2604.18575/) (2026)
- [Coreference as an indicator of context scope in multimodal narrative](https://mlsanity.com/paper/2503.05298/) (2025)
- [From Image Captioning to Visual Storytelling](https://mlsanity.com/paper/2508.14045/) (2025)
- [Unraveling the cognitive patterns of Large Language Models through module communities](https://mlsanity.com/paper/2508.18192/) (2025)
- [StoryMovie: A Dataset for Semantic Alignment of Visual Stories with Movie Scripts and Subtitles](https://mlsanity.com/paper/2602.21829/) (2026)

## Abstract (arXiv)

Current work on image-based story generation suffers from the fact that the existing image sequence collections do not have coherent plots behind them. We improve visual story generation by producing a new image-grounded dataset, Visual Writing Prompts (VWP). VWP contains almost 2K selected sequences of movie shots, each including 5-10 images. The image sequences are aligned with a total of 12K stories which were collected via crowdsourcing given the image sequences and a set of grounded characters from the corresponding image sequence. Our new image sequence collection and filtering process has allowed us to obtain stories that are more coherent and have more narrativity compared to previous work. We also propose a character-based story generation model driven by coherence as a strong baseline. Evaluations show that our generated stories are more coherent, visually grounded, and have more narrativity than stories generated with the current state-of-the-art model.