mlsanityData to 7 Oct 2026

Story Generation from Visual Inputs: Techniques, Related Tasks, and Challenges

arXiv 2406.02748 ↗First posted 4 Jun 2024cs.CV cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

Creating engaging narratives from visual data is crucial for automated digital media consumption, assistive technologies, and interactive entertainment. This survey covers methodologies used in the generation of these narratives, focusing on their principles, strengths, and limitations. The survey also covers tasks related to automatic story generation, such as image and video captioning, and visual question answering, as well as story generation without visual inputs. These tasks share common challenges with visual story generation and have served as inspiration for the techniques used in the field. We analyze the main datasets and evaluation metrics, providing a critical perspective on their limitations.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 56 of its 124 references that are in the map. First posted on arXiv 4 Jun 2024.

Builds on

56 of 124 references.

References · 56

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  9. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  10. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  11. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  12. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
38 more
  1. 2024Mixtral of Expertscited by 1,785
  2. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  3. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  4. 2018Hierarchical Neural Story Generationcited by 1,182
  5. 2021Video Swin Transformercited by 1,090
  6. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  7. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  8. 2017Dense-Captioning Events in Videoscited by 789
  9. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  10. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  11. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 484
  12. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  13. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 413
  14. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 201
  15. 2022Re3: Generating Longer Stories With Recursive Reprompting and Revisioncited by 193
  16. 2016Movie Descriptioncited by 180
  17. 2016Visual Storytellingcited by 176
  18. 2021SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioningcited by 161
  19. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 117
  20. 2015Describing Videos by Exploiting Temporal Structurecited by 88
  21. 2019Multimodal Transformer with Multi-View Visual Representation for Image Captioningcited by 86
  22. 2017DeepStory: Video Story QA by Deep Embedded Memory Networkscited by 67
  23. 2020STORIUM: A Dataset and Evaluation Platform for Machine-in-the-Loop Story Generationcited by 56
  24. 2018No Metrics Are Perfect: Adversarial Reward Learning for Visual Storytellingcited by 46
  25. 2019Story Realization: Expanding Plot Events into Sentencescited by 35
  26. 2017Show, Ask, Attend, and Answer: A Strong Baseline For Visual Question Answeringcited by 31
  27. 2018Adversarial Inference for Multi-Sentence Video Descriptioncited by 31
  28. 2023Visual Writing Prompts: Character-Grounded Story Generation with Curated Image Sequencescited by 31
  29. 2017Story Generation from Sequence of Independent Short Descriptionscited by 26
  30. 2019Knowledge-Enriched Visual Storytellingcited by 26
  31. 2018GLAC Net: GLocal Attention Cascading Networks for Multi-image Cued Story Generationcited by 23
  32. 2018Hierarchically Structured Reinforcement Learning for Topically Coherent Visual Story Generationcited by 21
  33. 2021Plot and Rework: Modeling Storylines for Visual Storytellingcited by 14
  34. 2018Contextualize, Show and Tell: A Neural Visual Storytellercited by 13
  35. 2020Hide-and-Tell: Learning to Bridge Photo Streams for Visual Storytellingcited by 13
  36. 2017Hierarchically-Attentive RNN for Album Summarization and Storytellingcited by 10
  37. 2019Visual Story Post-Editingcited by 10
  38. 2018A Pipeline for Creative Visual Storytellingcited by 7

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 1.

3 papers.

  1. 2025StoryReasoning Dataset: Using Chain-of-Thought for Scene Understanding and Grounded Story Generationcited by 3
  2. 2024Improving Visual Storytelling with Multimodal Large Language Modelscited by 2
  3. 2026StoryMovie: A Dataset for Semantic Alignment of Visual Stories with Movie Scripts and Subtitlescited by 0