# Not (yet) the whole story: Evaluating Visual Storytelling Requires More than Measuring Coherence, Grounding, and Repetition

arXiv: https://arxiv.org/abs/2407.04559 · HTML: https://mlsanity.com/paper/2407.04559/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 44 references that are in the map.
- First posted on arXiv 5 Jul 2024.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [ImageNet-21K Pretraining for the Masses](https://mlsanity.com/paper/2104.10972/) (2021)
- [MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers](https://mlsanity.com/paper/2102.01454/) (2021)
- [A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories](https://mlsanity.com/paper/1604.01696/) (2016)
- [Why We Need New Evaluation Metrics for NLG](https://mlsanity.com/paper/1707.06875/) (2017)
- [MovieNet: A Holistic Dataset for Movie Understanding](https://mlsanity.com/paper/2007.10937/) (2020)
- [Visual Storytelling](https://mlsanity.com/paper/1604.03968/) (2016)
- [What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variability](https://mlsanity.com/paper/2305.11707/) (2023)
- [No Metrics Are Perfect: Adversarial Reward Learning for Visual Storytelling](https://mlsanity.com/paper/1804.09160/) (2018)
- [Visual Writing Prompts: Character-Grounded Story Generation with Curated Image Sequences](https://mlsanity.com/paper/2301.08571/) (2023)
- [Knowledge-Enriched Visual Storytelling](https://mlsanity.com/paper/1912.01496/) (2019)
- [GLAC Net: GLocal Attention Cascading Networks for Multi-image Cued Story Generation](https://mlsanity.com/paper/1805.10973/) (2018)
- [Commonsense Knowledge Aware Concept Selection For Diverse and Informative Visual Storytelling](https://mlsanity.com/paper/2102.02963/) (2021)
- [Plot and Rework: Modeling Storylines for Visual Storytelling](https://mlsanity.com/paper/2105.06950/) (2021)
- [GROOViST: A Metric for Grounding Objects in Visual Storytelling](https://mlsanity.com/paper/2310.17770/) (2023)

## Built on by (most cited first, 5 of 5)

- [VIST-GPT: Ushering in the Era of Visual Storytelling with LLMs?](https://mlsanity.com/paper/2504.19267/) (2025)
- [Beyond Single Frames: Can LMMs Comprehend Temporal and Contextual Narratives in Image Sequences?](https://mlsanity.com/paper/2502.13925/) (2025)
- [Generating Visual Stories with Grounded and Coreferent Characters](https://mlsanity.com/paper/2409.13555/) (2024)
- [Natural Language Generation from Visual Events: State-of-the-Art and Key Open Questions](https://mlsanity.com/paper/2502.13034/) (2025)
- [Where is the multimodal goal post? On the Ability of Foundation Models to Recognize Highlights in Videos](https://mlsanity.com/paper/2601.16333/) (2026)

## Abstract (arXiv)

Visual storytelling consists in generating a natural language story given a temporally ordered sequence of images. This task is not only challenging for models, but also very difficult to evaluate with automatic metrics since there is no consensus about what makes a story 'good'. In this paper, we introduce a novel method that measures story quality in terms of human likeness regarding three key aspects highlighted in previous work: visual grounding, coherence, and repetitiveness. We then use this method to evaluate the stories generated by several models, showing that the foundation model LLaVA obtains the best result, but only slightly so compared to TAPM, a 50-times smaller visual storytelling model. Upgrading the visual and language components of TAPM results in a model that yields competitive performance with a relatively low number of parameters. Finally, we carry out a human evaluation study, whose results suggest that a 'good' story may require more than a human-like level of visual grounding, coherence, and repetition.