# RoViST:Learning Robust Metrics for Visual Storytelling

arXiv: https://arxiv.org/abs/2205.03774 · HTML: https://mlsanity.com/paper/2205.03774/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 43 references that are in the map.
- First posted on arXiv 8 May 2022.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [Unified Vision-Language Pre-Training for Image Captioning and VQA](https://mlsanity.com/paper/1909.11059/) (2019)
- [MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance](https://mlsanity.com/paper/1909.02622/) (2019)
- [A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories](https://mlsanity.com/paper/1604.01696/) (2016)
- [Stacked Cross Attention for Image-Text Matching](https://mlsanity.com/paper/1803.08024/) (2018)
- [Visual Storytelling](https://mlsanity.com/paper/1604.03968/) (2016)
- [UNION: An Unreferenced Metric for Evaluating Open-ended Story Generation](https://mlsanity.com/paper/2009.07602/) (2020)
- [TIGEr: Text-to-Image Grounding for Image Caption Evaluation](https://mlsanity.com/paper/1909.02050/) (2019)
- [Learning to Evaluate Image Captioning](https://mlsanity.com/paper/1806.06422/) (2018)
- [No Metrics Are Perfect: Adversarial Reward Learning for Visual Storytelling](https://mlsanity.com/paper/1804.09160/) (2018)
- [What Makes A Good Story? Designing Composite Rewards for Visual Storytelling](https://mlsanity.com/paper/1909.05316/) (2019)
- [Knowledge-Enriched Visual Storytelling](https://mlsanity.com/paper/1912.01496/) (2019)
- [GLAC Net: GLocal Attention Cascading Networks for Multi-image Cued Story Generation](https://mlsanity.com/paper/1805.10973/) (2018)
- [Commonsense Knowledge Aware Concept Selection For Diverse and Informative Visual Storytelling](https://mlsanity.com/paper/2102.02963/) (2021)
- [Visual Story Post-Editing](https://mlsanity.com/paper/1906.01764/) (2019)

## Built on by (most cited first, 7 of 7)

- [What Makes a Good Story and How Can We Measure It? A Comprehensive Survey of Story Evaluation](https://mlsanity.com/paper/2408.14622/) (2024)
- [VIST-GPT: Ushering in the Era of Visual Storytelling with LLMs?](https://mlsanity.com/paper/2504.19267/) (2025)
- [Visual Storytelling with Question-Answer Plans](https://mlsanity.com/paper/2310.05295/) (2023)
- [The Lost Melody: Empirical Observations on Text-to-Video Generation From A Storytelling Perspective](https://mlsanity.com/paper/2405.08720/) (2024)
- [Context-aware Visual Storytelling with Visual Prefix Tuning and Contrastive Learning](https://mlsanity.com/paper/2408.06259/) (2024)
- [Natural Language Generation from Visual Events: State-of-the-Art and Key Open Questions](https://mlsanity.com/paper/2502.13034/) (2025)
- [MMCOMET: A Large-Scale Multimodal Commonsense Knowledge Graph for Contextual Reasoning](https://mlsanity.com/paper/2603.01055/) (2026)

## Abstract (arXiv)

Visual storytelling (VST) is the task of generating a story paragraph that describes a given image sequence. Most existing storytelling approaches have evaluated their models using traditional natural language generation metrics like BLEU or CIDEr. However, such metrics based on n-gram matching tend to have poor correlation with human evaluation scores and do not explicitly consider other criteria necessary for storytelling such as sentence structure or topic coherence. Moreover, a single score is not enough to assess a story as it does not inform us about what specific errors were made by the model. In this paper, we propose 3 evaluation metrics sets that analyses which aspects we would look for in a good story: 1) visual grounding, 2) coherence, and 3) non-redundancy. We measure the reliability of our metric sets by analysing its correlation with human judgement scores on a sample of machine stories obtained from 4 state-of-the-arts models trained on the Visual Storytelling Dataset (VIST). Our metric sets outperforms other metrics on human correlation, and could be served as a learning based evaluation metric set that is complementary to existing rule-based metrics.