The Lost Melody: Empirical Observations on Text-to-Video Generation From A Storytelling Perspective
Not on the map. Why?
Abstract (arXiv)
Text-to-video generation task has witnessed a notable progress, with the generated outcomes reflecting the text prompts with high fidelity and impressive visual qualities. However, current text-to-video generation models are invariably focused on conveying the visual elements of a single scene, and have so far been indifferent to another important potential of the medium, namely a storytelling. In this paper, we examine text-to-video generation from a storytelling perspective, which has been hardly investigated, and make empirical remarks that spotlight the limitations of current text-to-video generation scheme. We also propose an evaluation framework for storytelling aspects of videos, and discuss the potential future directions.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 37 of its 50 references that are in the map. First posted on arXiv 13 May 2024.
Builds on
37 of 50 references.
References · 37
- 2017Attention Is All You Needcited by 59,932
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020Denoising Diffusion Probabilistic Modelscited by 20,851
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
- 2017Neural Discrete Representation Learningcited by 4,650
- 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
- 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
- 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
- 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
- 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
- 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
25 more
- 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
- 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
- 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 589
- 2018A Short Note about Kinetics-600cited by 331
- 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 305
- 2016Temporal Generative Adversarial Nets with Singular Value Clippingcited by 198
- 2024Fast Timing-Conditioned Latent Audio Diffusioncited by 178
- 2016Visual Storytellingcited by 176
- 2023VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planningcited by 138
- 2018StoryGAN: A Sequential Conditional GAN for Story Visualizationcited by 137
- 2018A Hierarchical Latent Vector Model for Learning Long-Term Structure in Musiccited by 131
- 2022StoryDALL-E: Adapting Pretrained Text-to-Image Transformers for Story Continuationcited by 74
- 2024VideoStudio: Generating Consistent-Content and Multi-Scene Videoscited by 62
- 2021OpenMEVA: A Benchmark for Evaluating Open-ended Story Generation Metricscited by 61
- 2020UNION: An Unreferenced Metric for Evaluating Open-ended Story Generationcited by 60
- 2018No Metrics Are Perfect: Adversarial Reward Learning for Visual Storytellingcited by 46
- 2017ObamaNet: Photo-realistic lip-sync from textcited by 42
- 2023OpenVoice: Versatile Instant Voice Cloningcited by 30
- 2019What Makes A Good Story? Designing Composite Rewards for Visual Storytellingcited by 29
- 2018Video Storytelling: Textual Summaries for Eventscited by 22
- 2022StoryER: Automatic Story Evaluation via Ranking, Rating and Reasoningcited by 21
- 2019Visual Story Post-Editingcited by 10
- 2022RoViST:Learning Robust Metrics for Visual Storytellingcited by 7
- 2023Visual Storytelling with Question-Answer Planscited by 5
- 2017PersonaBank: A Corpus of Personal Narratives and Their Story Intention Graphscited by 3
Built on by
Citing papers per year: 2024: 1; 2025: 1; 2026: 1.
3 papers.