mlsanityData to 7 Oct 2026

Intelligent Grimm -- Open-ended Visual Storytelling via Latent Diffusion Models

arXiv 2306.00973 ↗First posted 1 Jun 2023cs.CVcited by 76

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Generative models have recently exhibited exceptional capabilities in text-to-image generation, but still struggle to generate image sequences coherently. In this work, we focus on a novel, yet challenging task of generating a coherent image sequence based on a given storyline, denoted as open-ended visual storytelling. We make the following three contributions: (i) to fulfill the task of visual storytelling, we propose a learning-based auto-regressive image generation model, termed as StoryGen, with a novel vision-language context module, that enables to generate the current frame by conditioning on the corresponding text prompt and preceding image-caption pairs; (ii) to address the data shortage of visual storytelling, we collect paired image-text sequences by sourcing from online videos and open-source E-books, establishing processing pipeline for constructing a large-scale dataset with diverse characters, storylines, and artistic styles, named StorySalon; (iii) Quantitative experiments and human evaluations have validated the superiority of our StoryGen, where we show StoryGen can generalize to unseen characters without any optimization, and generate image sequences with coherent content and consistent character. Code, dataset, and models are available at https://haoningwu3639.github.io/StoryGen_Webpage/

Key facts. Cited by 76 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 57 of its 72 references that are in the map. First posted on arXiv 1 Jun 2023.

Builds on

57 of 72 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  2. 2022Synthesizing Coherent Story with Auto-Regressive Latent Diffusion Models20 Nov
  3. 2022InstructPix2Pix: Learning to Follow Image Editing Instructions17 Nov
  4. 2022Imagic: Text-Based Real Image Editing with Diffusion Models17 Oct
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  7. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec
  8. 2021Blended Diffusion for Text-driven Editing of Natural Images29 Nov

Other references · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  9. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  10. 2020Denoising Diffusion Implicit Modelscited by 9,655
  11. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  12. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
37 more
  1. 2022Classifier-Free Diffusion Guidancecited by 5,717
  2. 2022Generative Adversarial Networkscited by 5,603
  3. 2017Neural Discrete Representation Learningcited by 4,650
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  7. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  9. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  10. 2022Video Diffusion Modelscited by 2,280
  11. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  12. 2023Consistency Modelscited by 1,778
  13. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  14. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  15. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
  16. 2022YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectorscited by 978
  17. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  18. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
  19. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  20. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 589
  21. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 575
  22. 2023BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editingcited by 438
  23. 2023TEXTure: Text-Guided Texturing of 3D Shapescited by 303
  24. 2022SmartBrush: Text and Shape Guided Object Inpainting with Diffusion Modelcited by 276
  25. 2017StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 229
  26. 2016Visual Storytellingcited by 176
  27. 2023NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generationcited by 170
  28. 2018StoryGAN: A Sequential Conditional GAN for Story Visualizationcited by 137
  29. 2023Hierarchical Integration Diffusion Model for Realistic Image Deblurringcited by 87
  30. 2023ChatGPT Asks, BLIP-2 Answers: Automatic Questioning Towards Enriched Visual Descriptionscited by 86
  31. 2022StoryDALL-E: Adapting Pretrained Text-to-Image Transformers for Story Continuationcited by 74
  32. 2023TaleCrafter: Interactive Story Visualization with Multiple Characterscited by 71
  33. 2018Imagine This! Scripts to Compositions to Videoscited by 54
  34. 2021Integrating Visuospatial, Linguistic and Commonsense Structure into Story Visualizationcited by 51
  35. 2021Improving Generation and Evaluation of Visual Stories via Semantic Consistencycited by 43
  36. 2022Character-Centric Story Visualization via Visual Planning and Token Alignmentcited by 32
  37. 2022Word-Level Fine-Grained Story Visualizationcited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 28; 2025: 30; 2026: 13.

76 papers.

Sort:

By area:

  1. 2023WonderJourney: Going from Anywhere to Everywherecited by 157
  2. 2024In-Context LoRA for Diffusion Transformerscited by 155
  3. 2025One-Minute Video Generation with Test-Time Trainingcited by 100
  4. 2024MovieDreamer: Hierarchical Generation for Coherent Long Visual Sequencecited by 85
  5. 2025Automated Movie Generation via Multi-Agent CoT Planningcited by 82
  6. 2024SEED-Story: Multimodal Long Story Generation with Large Language Modelcited by 70
  7. 2025SceneGen: Single-Image 3D Scene Generation in One Feedforward Passcited by 49
  8. 2024DreamStory: Open-Domain Story Visualization by LLM-Guided Multi-Subject Consistent Diffusioncited by 46
  9. 2023AutoStory: Generating Diverse Storytelling Images with Minimal Human Effortcited by 38
  10. 2024MovieBench: A Hierarchical Movie Level Dataset for Long Video Generationcited by 35
  11. 2025StoryMem: Multi-shot Long Video Storytelling with Memorycited by 35
  12. 2024OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generationcited by 30
38 more
  1. 2024StoryAgent: Customized Storytelling Video Generation via Multi-Agent Collaborationcited by 28
  2. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  3. 2024M2Diffuser: Diffusion-based Trajectory Optimization for Mobile Manipulation in 3D Scenescited by 26
  4. 2023StoryGPT-V: Large Language Models as Consistent Story Visualizerscited by 25
  5. 2024MegaFusion: Extend Diffusion Models towards Higher-resolution Image Generation without Further Tuningcited by 25
  6. 2024CoMM: A Coherent Interleaved Image-Text Dataset for Multimodal Understanding and Generationcited by 21
  7. 2024StoryImager: A Unified and Efficient Framework for Coherent Story Visualization and Completioncited by 19
  8. 2024AutoStudio: Crafting Consistent Subjects in Multi-turn Interactive Image Generationcited by 19
  9. 2024Binarized Diffusion Model for Image Super-Resolutioncited by 18
  10. 2025MiLA: Multi-view Intensive-fidelity Long-term Video Generation World Model for Autonomous Drivingcited by 18
  11. 2025Generative AI for Cel-Animation: A Surveycited by 17
  12. 2023OpenLEAF: Open-Domain Interleaved Image-Text Generation and Evaluationcited by 16
  13. 2024DiffSensei: Bridging Multi-Modal LLMs and Diffusion Models for Customized Manga Generationcited by 16
  14. 2025Story2Board: A Training-Free Approach for Expressive Storyboard Generationcited by 16
  15. 2024Unbounded: A Generative Infinite Game of Character Life Simulationcited by 14
  16. 2024Story-Iter: A Training-free Iterative Paradigm for Long Story Visualizationcited by 12
  17. 2024Replication in Visual Diffusion Models: A Survey and Outlookcited by 11
  18. 2024ContextualStory: Consistent Visual Storytelling with Spatially-Enhanced and Storyline Contextcited by 8
  19. 2023Make-A-Storyboard: A General Framework for Storyboard with Disentangled and Merged Controlcited by 7
  20. 2024CinePreGen: Camera Controllable Video Previsualization via Engine-powered Diffusioncited by 7
  21. 2025Object Isolated Attention for Consistent Story Visualizationcited by 7
  22. 2025SceneDecorator: Towards Scene-Oriented Story Generation with Scene Planning and Scene Consistencycited by 7
  23. 2024Many-to-many Image Generation with Auto-regressive Diffusion Modelscited by 6
  24. 2024Prompt Tuning with Diffusion for Few-Shot Pre-trained Policy Generalizationcited by 6
  25. 2025Lay2Story: Extending Diffusion Transformers for Layout-Togglable Story Generationcited by 6
  26. 2026CutClaw: Agentic Hours-Long Video Editing via Music Synchronizationcited by 6
  27. 2026InfinityStory: Unlimited Video Generation with World Consistency and Character-Aware Shot Transitionscited by 5
  28. 2024RefDrop: Controllable Consistency in Image or Video Generation via Reference Feature Guidancecited by 4
  29. 2025VisAgent: Narrative-Preserving Story Visualization Frameworkcited by 4
  30. 2025VinaBench: Benchmark for Faithful and Consistent Visual Narrativescited by 4
  31. 2025Storybooth: Training-free Multi-Subject Consistency for Improved Visual Storytellingcited by 4
  32. 2024Openstory++: A Large-scale Dataset and Benchmark for Instance-aware Open-domain Visual Storytellingcited by 3
  33. 2024Improving Multi-Subject Consistency in Open-Domain Image Generation with Isolation and Reposition Attentioncited by 3
  34. 2024MRGen: Segmentation Data Engine for Underrepresented MRI Modalitiescited by 3
  35. 2024Motion by Queries: Identity-Motion Trade-offs in Text-to-Video Generationcited by 3
  36. 2025A High-Quality Dataset and Reliable Evaluation for Interleaved Image-Text Generationcited by 3
  37. 2025ViSTA: Visual Storytelling using Multi-modal Adapters for Text-to-Image Diffusion Modelscited by 3
  38. 2025TreeQ: Pushing the Quantization Boundary of Diffusion Transformer via Tree-Structured Mixed-Precision Searchcited by 3

50 most cited of 76 All 76 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Zero-shot Generation of Coherent Storybook from Plain Text Story using Diffusion Models
  2. 2023Improved Visual Story Generation with Adaptive Context Modeling
  3. 2024Evolving Storytelling: Benchmarks and Methods for New Character Customization with Diffusion Models
  4. 2025Generating Storytelling Images with Rich Chains-of-Reasoning
  5. 2023MagicScroll: Nontypical Aspect-Ratio Image Generation for Visual Storytelling via Multi-Layered Semantic-Aware Denoising
  6. 2026DATAREEL: Automated Data-Driven Video Story Generation with Animations
  7. 2024Imagining from Images with an AI Storytelling Tool
  8. 2026DreamShot: Personalized Storyboard Synthesis with Video Diffusion Prior