mlsanityData to 7 Oct 2026

StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation

arXiv 2405.01434 ↗First posted 2 May 2024cs.CVcited by 240

On the map: Diffusion models → Video generation

Abstract (arXiv)

For recent diffusion-based generative models, maintaining consistent content across a series of generated images, especially those containing subjects and complex details, presents a significant challenge. In this paper, we propose a new way of self-attention calculation, termed Consistent Self-Attention, that significantly boosts the consistency between the generated images and augments prevalent pretrained diffusion-based text-to-image models in a zero-shot manner. To extend our method to long-range video generation, we further introduce a novel semantic space temporal motion prediction module, named Semantic Motion Predictor. It is trained to estimate the motion conditions between two provided images in the semantic spaces. This module converts the generated sequence of images into videos with smooth transitions and consistent subjects that are significantly more stable than the modules based on latent spaces only, especially in the context of long video generation. By merging these two novel components, our framework, referred to as StoryDiffusion, can describe a text-based story with consistent images or videos encompassing a rich variety of contents. The proposed StoryDiffusion encompasses pioneering explorations in visual story generation with the presentation of images and videos, which we hope could inspire more research from the aspect of architectural modifications. Our code is made publicly available at https://github.com/HVision-NKU/StoryDiffusion.

Key facts. Cited by 240 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 68 of its 70 references that are in the map. First posted on arXiv 2 May 2024.

Builds on

68 of 70 references.

Closest ancestors · 17

Same direction, newest first

  1. 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation28 Nov
  2. 2023SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Models28 Nov
  3. 2023MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model27 Nov
  4. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov
  5. 2023SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Prediction31 Oct
  6. 2023LAMP: Learn A Motion Pattern for Few-Shot-Based Video Generation16 Oct
  7. 2023MotionDirector: Motion Customization of Text-to-Video Diffusion Models12 Oct
  8. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation27 Sep
  9. 2023ModelScope Text-to-Video Technical Report12 Aug
  10. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning10 Jul
  11. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
  12. 2023Conditional Image-to-Video Generation with Latent Flow Diffusion Models24 Mar
5 more
  1. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators23 Mar
  2. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation22 Dec
  3. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Models20 Nov
  4. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  5. 2022Video Diffusion Models7 Apr

Other references · 51

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  11. 2022Scalable Diffusion Models with Transformerscited by 5,931
  12. 2022Classifier-Free Diffusion Guidancecited by 5,717
38 more
  1. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  2. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  3. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  4. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  5. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  6. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
  7. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  8. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
  9. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
  10. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
  11. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  12. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  13. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  14. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  15. 2023MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editingcited by 722
  16. 2022LION: Latent Point Diffusion Models for 3D Shape Generationcited by 552
  17. 2022Fast Sampling of Diffusion Models with Exponential Integratorcited by 526
  18. 20213D Shape Generation and Completion through Point-Voxel Diffusioncited by 516
  19. 2023Exploiting Diffusion Prior for Real-World Image Super-Resolutioncited by 486
  20. 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 440
  21. 2024InstantID: Zero-shot Identity-Preserving Generation in Secondscited by 431
  22. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesiscited by 386
  23. 2023PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embeddingcited by 324
  24. 2023Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videoscited by 312
  25. 2021SegDiff: Image Segmentation with Diffusion Probabilistic Modelscited by 279
  26. 2022SmartBrush: Text and Shape Guided Object Inpainting with Diffusion Modelcited by 276
  27. 2022Improving Sample Quality of Diffusion Models Using Self-Attention Guidancecited by 151
  28. 2023Diffuse, Attend, and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusioncited by 131
  29. 2024Training-Free Consistent Text-to-Image Generationcited by 123
  30. 2023Animate-A-Story: Storytelling with Retrieval-Augmented Video Generationcited by 89
  31. 2023Open-vocabulary Object Segmentation with Diffusion Modelscited by 82
  32. 2023Directed Diffusion: Direct Control of Object Placement through Attention Guidancecited by 69
  33. 2023The Chosen One: Consistent Characters in Text-to-Image Diffusion Modelscited by 62
  34. 2023Text2Performer: Text-Driven Human Video Generationcited by 53
  35. 2023DirecT2V: Large Language Models are Frame-Level Directors for Zero-Shot Text-to-Video Generationcited by 47
  36. 2023Probabilistic Adaptation of Text-to-Video Modelscited by 29
  37. 2023Zero-shot Generation of Coherent Storybook from Plain Text Story using Diffusion Modelscited by 26
  38. 2023MaskDiffusion: Boosting Text-to-Image Consistency with Conditional Maskcited by 17

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 48; 2025: 116; 2026: 76.

240 papers.

Sort:

By area:

  1. 2024In-Context LoRA for Diffusion Transformerscited by 155
  2. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 154
  3. 2025Emu3.5: Native Multimodal Models are World Learnerscited by 119
  4. 2025Go-with-the-Flow: Motion-Controllable Video Diffusion Models Using Real-Time Warped Noisecited by 114
  5. 2025Long Context Tuning for Video Generationcited by 110
  6. 2024Adaptive Caching for Faster Video Generation with Diffusion Transformerscited by 100
  7. 2025One-Minute Video Generation with Test-Time Trainingcited by 100
  8. 2024Hallo2: Long-Duration and High-Resolution Audio-Driven Portrait Image Animationcited by 95
  9. 2024MotionBooth: Motion-Aware Customized Text-to-Video Generationcited by 92
  10. 2024DeMamba: AI-Generated Video Detection on Million-Scale GenVideo Benchmarkcited by 85
  11. 2024MovieDreamer: Hierarchical Generation for Coherent Long Visual Sequencecited by 85
  12. 2025Automated Movie Generation via Multi-Agent CoT Planningcited by 82
38 more
  1. 2025SkyReels-A2: Compose Anything in Video Diffusion Transformerscited by 72
  2. 2025A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluationscited by 66
  3. 2024Progressive Autoregressive Video Diffusion Modelscited by 65
  4. 2025Multi-subject Open-set Personalization in Video Generationcited by 65
  5. 2026Helios: Real Real-Time Long Video Generation Modelcited by 56
  6. 2025HoloCine: Holistic Generation of Cinematic Multi-Shot Long Video Narrativescited by 51
  7. 2025Captain Cinema: Towards Short Movie Generationcited by 48
  8. 2024DreamStory: Open-Domain Story Visualization by LLM-Guided Multi-Subject Consistent Diffusioncited by 46
  9. 2025MemFlow: Flowing Adaptive Memory for Consistent and Efficient Long Video Narrativescited by 40
  10. 2024Mind the Time: Temporally-Controlled Multi-Event Video Generationcited by 39
  11. 2024Looking Backward: Streaming Video-to-Video Translation with Feature Bankscited by 38
  12. 2025DICEPTION: A Generalist Diffusion Model for Visual Perceptual Taskscited by 37
  13. 2025ShotAdapter: Text-to-Multi-Shot Video Generation with Diffusion Modelscited by 37
  14. 2024MovieBench: A Hierarchical Movie Level Dataset for Long Video Generationcited by 35
  15. 2025StoryMem: Multi-shot Long Video Storytelling with Memorycited by 35
  16. 2025VideoGen-of-Thought: Step-by-step generating multi-shot video with minimal manual interventioncited by 33
  17. 2025MultiShotMaster: A Controllable Multi-Shot Video Generation Frameworkcited by 33
  18. 2025MoCha: Towards Movie-Grade Talking Character Synthesiscited by 31
  19. 2025OneStory: Coherent Multi-Shot Video Generation with Adaptive Memorycited by 31
  20. 2024Video-Infinity: Distributed Long Video Generationcited by 29
  21. 2024StoryAgent: Customized Storytelling Video Generation via Multi-Agent Collaborationcited by 28
  22. 2025EchoShot: Multi-Shot Portrait Video Generationcited by 28
  23. 2024StoryMaker: Towards Holistic Consistent Characters in Text-to-image Generationcited by 27
  24. 2025CineTrans: Learning to Generate Videos with Cinematic Transitions via Masked Diffusion Modelscited by 26
  25. 2024SnapGen-V: Generating a Five-Second Video within Five Seconds on a Mobile Devicecited by 25
  26. 2025Generating Multi-Image Synthetic Data for Text-to-Image Customizationcited by 25
  27. 2025MM-StoryAgent: Immersive Narrated Storybook Video Generation with a Multi-Agent Paradigm across Text, Image and Audiocited by 24
  28. 2025Mask2DiT: Dual Mask-based Diffusion Transformer for Multi-Scene Long Video Generationcited by 24
  29. 2025DreamRelation: Relation-Centric Video Customizationcited by 23
  30. 2025Code2Video: A Code-centric Paradigm for Educational Video Generationcited by 23
  31. 2024Group Diffusion Transformers are Unsupervised Multitask Learnerscited by 22
  32. 2025Personalize Anything for Free with Diffusion Transformercited by 22
  33. 2025MAGREF: Masked Guidance for Any-Reference Video Generation with Subject Disentanglementcited by 22
  34. 2026VideoMemory: Toward Consistent Video Generation via Memory Integrationcited by 22
  35. 2026ShotStream: Streaming Multi-Shot Video Generation for Interactive Storytellingcited by 22
  36. 2025NarrLV: Towards a Comprehensive Narrative-Centric Evaluation for Long Video Generationcited by 21
  37. 2024Towards Precise Scaling Laws for Video Diffusion Transformerscited by 20
  38. 2025Dynamic Concepts Personalization from Single Videoscited by 20

50 most cited of 240 All 240 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Boosting Consistency in Story Visualization with Rich-Contextual Conditional Diffusion Models
  2. 2024Animate Your Motion: Turning Still Images into Dynamic Videos
  3. 2025SemanticGen: Video Generation in Semantic Space
  4. 2025Pushing the Boundaries of State Space Models for Image and Video Generation
  5. 2026DCDM: Divide-and-Conquer Diffusion Models for Consistency-Preserving Video Generation
  6. 2025RepVideo: Rethinking Cross-Layer Representation for Video Generation
  7. 2024VideoTetris: Towards Compositional Text-to-Video Generation
  8. 2023LatentWarp: Consistent Diffusion Latents for Zero-Shot Video-to-Video Translation