# What Makes A Good Story? Designing Composite Rewards for Visual Storytelling

arXiv: https://arxiv.org/abs/1909.05316 · HTML: https://mlsanity.com/paper/1909.05316/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Sep 2019.

## Built on by (most cited first, 29 of 29)

- [Commonsense Knowledge Aware Concept Selection For Diverse and Informative Visual Storytelling](https://mlsanity.com/paper/2102.02963/) (2021)
- [Plot and Rework: Modeling Storylines for Visual Storytelling](https://mlsanity.com/paper/2105.06950/) (2021)
- [A Survey of Natural Language Generation](https://mlsanity.com/paper/2112.11739/) (2021)
- [Evaluating Document Coherence Modelling](https://mlsanity.com/paper/2103.10133/) (2021)
- [GROOViST: A Metric for Grounding Objects in Visual Storytelling](https://mlsanity.com/paper/2310.17770/) (2023)
- [RoViST:Learning Robust Metrics for Visual Storytelling](https://mlsanity.com/paper/2205.03774/) (2022)
- [TeViS:Translating Text Synopses to Video Storyboards](https://mlsanity.com/paper/2301.00135/) (2022)
- [Visual Storytelling with Question-Answer Plans](https://mlsanity.com/paper/2310.05295/) (2023)
- [SCO-VIST: Social Interaction Commonsense Knowledge-based Visual Storytelling](https://mlsanity.com/paper/2402.00319/) (2024)
- [HL Dataset: Visually-grounded Description of Scenes, Actions and Rationales](https://mlsanity.com/paper/2302.12189/) (2023)
- [Let Storytelling Tell Vivid Stories: An Expressive and Fluent Multimodal Storyteller](https://mlsanity.com/paper/2403.07301/) (2024)
- [Nested-Wasserstein Self-Imitation Learning for Sequence Generation](https://mlsanity.com/paper/2001.06944/) (2020)
- [Knowledge-enriched Attention Network with Group-wise Semantic for Visual Storytelling](https://mlsanity.com/paper/2203.05346/) (2022)
- [The Lost Melody: Empirical Observations on Text-to-Video Generation From A Storytelling Perspective](https://mlsanity.com/paper/2405.08720/) (2024)
- [Vision Transformer Based Model for Describing a Set of Images as a Story](https://mlsanity.com/paper/2210.02762/) (2022)
- [Multi-VQG: Generating Engaging Questions for Multiple Images](https://mlsanity.com/paper/2211.07441/) (2022)
- [Fine-grained length controllable video captioning with ordinal embeddings](https://mlsanity.com/paper/2408.15447/) (2024)
- [Local and Global Context-Based Pairwise Models for Sentence Ordering](https://mlsanity.com/paper/2110.04291/) (2021)
- [Affective Feedback Synthesis Towards Multimodal Text and Image Data](https://mlsanity.com/paper/2203.12692/) (2022)
- [Open-Domain Text Evaluation via Contrastive Distribution Methods](https://mlsanity.com/paper/2306.11879/) (2023)
- [Context-aware Visual Storytelling with Visual Prefix Tuning and Contrastive Learning](https://mlsanity.com/paper/2408.06259/) (2024)
- [Generating Visual Stories with Grounded and Coreferent Characters](https://mlsanity.com/paper/2409.13555/) (2024)
- [Weaving Visual Narratives: Agentic Image Bundle Composition Beyond Atomic Visual Matching](https://mlsanity.com/paper/2608.28695/) (2026)
- [Pruned Graph Neural Network for Short Story Ordering](https://mlsanity.com/paper/2203.06778/) (2022)
- [DiffuVST: Narrating Fictional Scenes with Global-History-Guided Denoising Models](https://mlsanity.com/paper/2312.07066/) (2023)
- [TARN-VIST: Topic Aware Reinforcement Network for Visual Storytelling](https://mlsanity.com/paper/2403.11550/) (2024)
- [Mining Contextualized Visual Associations from Images for Creativity Understanding](https://mlsanity.com/paper/2507.18915/) (2025)
- [From Image Captioning to Visual Storytelling](https://mlsanity.com/paper/2508.14045/) (2025)
- [Reviewing the Reviewer: LLM-Assisted Reviewer Feedback Generation for Guideline Compliance](https://mlsanity.com/paper/2602.10118/) (2026)

## Abstract (arXiv)

Previous storytelling approaches mostly focused on optimizing traditional metrics such as BLEU, ROUGE and CIDEr. In this paper, we re-examine this problem from a different angle, by looking deep into what defines a realistically-natural and topically-coherent story. To this end, we propose three assessment criteria: relevance, coherence and expressiveness, which we observe through empirical analysis could constitute a "high-quality" story to the human eye. Following this quality guideline, we propose a reinforcement learning framework, ReCo-RL, with reward functions designed to capture the essence of these quality criteria. Experiments on the Visual Storytelling Dataset (VIST) with both automatic and human evaluations demonstrate that our ReCo-RL model achieves better performance than state-of-the-art baselines on both traditional metrics and the proposed new criteria.