# Knowledge-Enriched Visual Storytelling

arXiv: https://arxiv.org/abs/1912.01496 · HTML: https://mlsanity.com/paper/1912.01496/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Dec 2019.

## Built on by (most cited first, 26 of 26)

- [Multi-Modal Knowledge Graph Construction and Application: A Survey](https://mlsanity.com/paper/2202.05786/) (2022)
- [Visual Writing Prompts: Character-Grounded Story Generation with Curated Image Sequences](https://mlsanity.com/paper/2301.08571/) (2023)
- [Commonsense Knowledge Aware Concept Selection For Diverse and Informative Visual Storytelling](https://mlsanity.com/paper/2102.02963/) (2021)
- [Plot and Rework: Modeling Storylines for Visual Storytelling](https://mlsanity.com/paper/2105.06950/) (2021)
- [A survey on knowledge-enhanced multimodal learning](https://mlsanity.com/paper/2211.12328/) (2022)
- [VIST-GPT: Ushering in the Era of Visual Storytelling with LLMs?](https://mlsanity.com/paper/2504.19267/) (2025)
- [RoViST:Learning Robust Metrics for Visual Storytelling](https://mlsanity.com/paper/2205.03774/) (2022)
- [Ordered Attention for Coherent Visual Storytelling](https://mlsanity.com/paper/2108.02180/) (2021)
- [Detecting and Grounding Important Characters in Visual Stories](https://mlsanity.com/paper/2303.17647/) (2023)
- [Visual Storytelling with Question-Answer Plans](https://mlsanity.com/paper/2310.05295/) (2023)
- [SCO-VIST: Social Interaction Commonsense Knowledge-based Visual Storytelling](https://mlsanity.com/paper/2402.00319/) (2024)
- [Not (yet) the whole story: Evaluating Visual Storytelling Requires More than Measuring Coherence, Grounding, and Repetition](https://mlsanity.com/paper/2407.04559/) (2024)
- [Let Storytelling Tell Vivid Stories: An Expressive and Fluent Multimodal Storyteller](https://mlsanity.com/paper/2403.07301/) (2024)
- [Knowledge-enriched Attention Network with Group-wise Semantic for Visual Storytelling](https://mlsanity.com/paper/2203.05346/) (2022)
- [Text-Only Training for Visual Storytelling](https://mlsanity.com/paper/2308.08881/) (2023)
- [Story Generation from Visual Inputs: Techniques, Related Tasks, and Challenges](https://mlsanity.com/paper/2406.02748/) (2024)
- [The Contribution of Knowledge in Visiolinguistic Learning: A Survey on Tasks and Challenges](https://mlsanity.com/paper/2303.02411/) (2023)
- [Narrative-to-Scene Generation: An LLM-Driven Pipeline for 2D Game Environments](https://mlsanity.com/paper/2509.04481/) (2025)
- [Context-aware Visual Storytelling with Visual Prefix Tuning and Contrastive Learning](https://mlsanity.com/paper/2408.06259/) (2024)
- [Generating Visual Stories with Grounded and Coreferent Characters](https://mlsanity.com/paper/2409.13555/) (2024)
- [Natural Language Generation from Visual Events: State-of-the-Art and Key Open Questions](https://mlsanity.com/paper/2502.13034/) (2025)
- [Robust Symbolic Reasoning for Visual Narratives via Hierarchical and Semantically Normalized Knowledge Graphs](https://mlsanity.com/paper/2508.14941/) (2025)
- [Let's Talk! Striking Up Conversations via Conversational Visual Question Generation](https://mlsanity.com/paper/2205.09327/) (2022)
- [DiffuVST: Narrating Fictional Scenes with Global-History-Guided Denoising Models](https://mlsanity.com/paper/2312.07066/) (2023)
- [TARN-VIST: Topic Aware Reinforcement Network for Visual Storytelling](https://mlsanity.com/paper/2403.11550/) (2024)
- [From Image Captioning to Visual Storytelling](https://mlsanity.com/paper/2508.14045/) (2025)

## Abstract (arXiv)

Stories are diverse and highly personalized, resulting in a large possible output space for story generation. Existing end-to-end approaches produce monotonous stories because they are limited to the vocabulary and knowledge in a single training dataset. This paper introduces KG-Story, a three-stage framework that allows the story generation model to take advantage of external Knowledge Graphs to produce interesting stories. KG-Story distills a set of representative words from the input prompts, enriches the word set by using external knowledge graphs, and finally generates stories based on the enriched word set. This distill-enrich-generate framework allows the use of external resources not only for the enrichment phase, but also for the distillation and generation phases. In this paper, we show the superiority of KG-Story for visual storytelling, where the input prompt is a sequence of five photos and the output is a short story. Per the human ranking evaluation, stories generated by KG-Story are on average ranked better than that of the state-of-the-art systems. Our code and output stories are available at https://github.com/zychen423/KE-VIST.