mlsanityData to 14 Oct 2026

Animate-A-Story: Storytelling with Retrieval-Augmented Video Generation

arXiv 2307.06940 ↗First posted 13 Jul 2023cs.CVcited by 90

Not on the map. Why?

Cited by 31 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Generating videos for visual storytelling can be a tedious and complex process that typically requires either live-action filming or graphics animation rendering. To bypass these challenges, our key idea is to utilize the abundance of existing video clips and synthesize a coherent storytelling video by customizing their appearances. We achieve this by developing a framework comprised of two functional modules: (i) Motion Structure Retrieval, which provides video candidates with desired scene or motion context described by query texts, and (ii) Structure-Guided Text-to-Video Synthesis, which generates plot-aligned videos under the guidance of motion structure and text prompts. For the first module, we leverage an off-the-shelf video retrieval system and extract video depths as motion structure. For the second module, we propose a controllable video generation model that offers flexible controls over structure and characters. The videos are synthesized by following the structural guidance and appearance instruction. To ensure visual consistency across clips, we propose an effective concept personalization approach, which allows the specification of the desired character identities through text prompts. Extensive experiments demonstrate that our approach exhibits significant advantages over various existing baselines.

Key facts. Cited by 90 papers in the map (data through 14 Oct 2026). Builds on 42 of its 46 references that are in the map. First posted on arXiv 13 Jul 2023.

Builds on

42 of 46 references.

References · 42

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  8. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  9. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  10. 2022Video Diffusion Modelscited by 2,305
  11. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  12. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
30 more
  1. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  2. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,495
  3. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
  4. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  5. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,106
  6. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  7. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 670
  8. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 658
  9. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 524
  10. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 484
  11. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 465
  12. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 464
  13. 2023ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generationcited by 416
  14. 2016Generating Videos with Scene Dynamicscited by 376
  15. 2022MAGVIT: Masked Generative Video Transformercited by 346
  16. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generationcited by 331
  17. 2023ControlVideo: Training-free Controllable Text-to-Video Generationcited by 327
  18. 2023Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videoscited by 314
  19. 2023Composer: Creative and Controllable Image Synthesis with Composable Conditionscited by 304
  20. 2021StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2cited by 301
  21. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 256
  22. 2023Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translationcited by 253
  23. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  24. 2016Temporal Generative Adversarial Nets with Singular Value Clippingcited by 198
  25. 2022Sketch-Guided Text-to-Image Diffusion Modelscited by 180
  26. 2023Video Probabilistic Diffusion Models in Projected Latent Spacecited by 171
  27. 2023Gen-L-Video: Multi-Text to Long Video Generation via Temporal Co-Denoisingcited by 139
  28. 2023Make-Your-Video: Customized Video Generation Using Textual and Structural Guidancecited by 129
  29. 2023A Neural Space-Time Representation for Text-to-Image Personalizationcited by 123
  30. 2022VIDM: Video Implicit Diffusion Modelscited by 78

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 11; 2024: 36; 2025: 33; 2026: 10.

90 papers.

Sort:

By area:

  1. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  2. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,660
  3. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 477
  4. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 354
  5. 2025VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulnesscited by 315
  6. 2024StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generationcited by 247
  7. 2023A Survey on Video Diffusion Modelscited by 206
  8. 2023FreeNoise: Tuning-Free Longer Video Diffusion via Noise Reschedulingcited by 180
  9. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 159
  10. 2023VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planningcited by 141
  11. 2025Long Context Tuning for Video Generationcited by 113
  12. 2024Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Alignerscited by 111
38 more
  1. 2024VideoDPO: Omni-Preference Alignment for Video Diffusion Generationcited by 108
  2. 2023Generative Image Dynamicscited by 94
  3. 2024MovieDreamer: Hierarchical Generation for Coherent Long Visual Sequencecited by 89
  4. 2024Video as the New Language for Real-World Decision Makingcited by 87
  5. 2023InstructVideo: Instructing Video Diffusion Models with Human Feedbackcited by 86
  6. 2024Vlogger: Make Your Dream A Vlogcited by 70
  7. 2023MagicStick: Controllable Video Editing via Control Handle Transformationscited by 62
  8. 2024CustomCrafter: Customized Video Generation with Preserving Motion and Concept Composition Abilitiescited by 60
  9. 2024FreeTraj: Tuning-Free Trajectory Control in Video Diffusion Modelscited by 56
  10. 2024Follow-Your-Click: Open-domain Regional Image Animation via Short Promptscited by 52
  11. 2025Captain Cinema: Towards Short Movie Generationcited by 48
  12. 2024PhysMotion: Physics-Grounded Dynamics From a Single Imagecited by 45
  13. 2025PhysAnimator: Physics-Guided Generative Cartoon Animationcited by 42
  14. 2024Efficient Diffusion Models: A Comprehensive Survey from Principles to Practicescited by 41
  15. 2023MEVG: Multi-event Video Generation with Text-to-Video Modelscited by 40
  16. 2024Mind the Time: Temporally-Controlled Multi-Event Video Generationcited by 40
  17. 2024LeviTor: 3D Trajectory Oriented Image-to-Video Synthesiscited by 39
  18. 2025VideoGen-of-Thought: Step-by-step generating multi-shot video with minimal manual interventioncited by 36
  19. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  20. 2025Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate Video Diffusion Transformerscited by 34
  21. 2025Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlookcited by 32
  22. 2024TRIP: Temporal Residual Learning with Image Noise Prior for Image-to-Video Diffusion Modelscited by 31
  23. 2025Exploring the Evolution of Physics Cognition in Video Generation: A Surveycited by 31
  24. 2024A Survey on Personalized Content Synthesis with Diffusion Modelscited by 29
  25. 2024PersonalVideo: High ID-Fidelity Video Customization without Dynamic and Semantic Degradationcited by 29
  26. 2025CineTrans: Learning to Generate Videos with Cinematic Transitions via Masked Diffusion Modelscited by 29
  27. 2024RAG and RAU: A Survey on Retrieval-Augmented Language Model in Natural Language Processingcited by 25
  28. 2025OmniVDiff: Omni Controllable Video Diffusion for Generation and Understandingcited by 22
  29. 2025The Devil is in the Prompts: Retrieval-Augmented Prompt Optimization for Text-to-Video Generationcited by 21
  30. 2025PresentAgent: Multimodal Agent for Presentation Video Generationcited by 21
  31. 2023MotionCrafter: One-Shot Motion Customization of Diffusion Modelscited by 20
  32. 2024Language-Guided Joint Audio-Visual Editing via One-Shot Adaptationcited by 19
  33. 2024Large Motion Video Autoencoding with Cross-modal Video VAEcited by 19
  34. 2023Fine-grained Controllable Video Generation via Object Appearance and Contextcited by 18
  35. 2025VideoAuteur: Towards Long Narrative Video Generationcited by 18
  36. 2025Generative AI for Cel-Animation: A Surveycited by 17
  37. 2024VideoMaker: Zero-shot Customized Video Generation with the Inherent Force of Video Diffusion Modelscited by 16
  38. 2025Model Reveals What to Cache: Profiling-Based Feature Reuse for Video Diffusion Modelscited by 15

50 most cited of 90 All 90 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Complementary Retrieval-Augmented Prompting for Consistent Long-Form Video Generation
  2. 2024Synchronized Video Storytelling: Generating Video Narrations with Structured Storyline
  3. 2026DreamShot: Personalized Storyboard Synthesis with Video Diffusion Prior
  4. 2025NarrLV: Towards a Comprehensive Narrative-Centric Evaluation for Long Video Generation
  5. 2026DATAREEL: Automated Data-Driven Video Story Generation with Animations
  6. 2026Customized Visual Storytelling with Unified Multimodal LLMs
  7. 2025FairyGen: Storied Cartoon Video from a Single Child-Drawn Character
  8. 2023Intelligent Grimm -- Open-ended Visual Storytelling via Latent Diffusion Models