mlsanityData to 7 Oct 2026

Not (yet) the whole story: Evaluating Visual Storytelling Requires More than Measuring Coherence, Grounding, and Repetition

arXiv 2407.04559 ↗First posted 5 Jul 2024cs.CL cs.AI cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

Visual storytelling consists in generating a natural language story given a temporally ordered sequence of images. This task is not only challenging for models, but also very difficult to evaluate with automatic metrics since there is no consensus about what makes a story 'good'. In this paper, we introduce a novel method that measures story quality in terms of human likeness regarding three key aspects highlighted in previous work: visual grounding, coherence, and repetitiveness. We then use this method to evaluate the stories generated by several models, showing that the foundation model LLaVA obtains the best result, but only slightly so compared to TAPM, a 50-times smaller visual storytelling model. Upgrading the visual and language components of TAPM results in a model that yields competitive performance with a relatively low number of parameters. Finally, we carry out a human evaluation study, whose results suggest that a 'good' story may require more than a human-like level of visual grounding, coherence, and repetition.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 29 of its 44 references that are in the map. First posted on arXiv 5 Jul 2024.

Builds on

29 of 44 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  10. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  11. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  12. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
17 more
  1. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  2. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  3. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  4. 2021ImageNet-21K Pretraining for the Massescited by 607
  5. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  6. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 413
  7. 2017Why We Need New Evaluation Metrics for NLGcited by 232
  8. 2020MovieNet: A Holistic Dataset for Movie Understandingcited by 206
  9. 2016Visual Storytellingcited by 176
  10. 2023What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variabilitycited by 51
  11. 2018No Metrics Are Perfect: Adversarial Reward Learning for Visual Storytellingcited by 46
  12. 2023Visual Writing Prompts: Character-Grounded Story Generation with Curated Image Sequencescited by 31
  13. 2019Knowledge-Enriched Visual Storytellingcited by 26
  14. 2018GLAC Net: GLocal Attention Cascading Networks for Multi-image Cued Story Generationcited by 23
  15. 2021Commonsense Knowledge Aware Concept Selection For Diverse and Informative Visual Storytellingcited by 22
  16. 2021Plot and Rework: Modeling Storylines for Visual Storytellingcited by 14
  17. 2023GROOViST: A Metric for Grounding Objects in Visual Storytellingcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 1.

5 papers.

  1. 2025VIST-GPT: Ushering in the Era of Visual Storytelling with LLMs?cited by 11
  2. 2025Beyond Single Frames: Can LMMs Comprehend Temporal and Contextual Narratives in Image Sequences?cited by 4
  3. 2024Generating Visual Stories with Grounded and Coreferent Characterscited by 1
  4. 2025Natural Language Generation from Visual Events: State-of-the-Art and Key Open Questionscited by 1
  5. 2026Where is the multimodal goal post? On the Ability of Foundation Models to Recognize Contextually Important Momentscited by 0