# Boosting Consistency in Story Visualization with Rich-Contextual Conditional Diffusion Models

arXiv: https://arxiv.org/abs/2407.02482 · HTML: https://mlsanity.com/paper/2407.02482/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 41 of its 44 references that are in the map.
- First posted on arXiv 2 Jul 2024.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 41)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023)
- [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023)
- [An Introduction to Variational Autoencoders](https://mlsanity.com/paper/1906.02691/) (2019)
- [Self-Attention Generative Adversarial Networks](https://mlsanity.com/paper/1805.08318/) (2018)
- [Autoregressive Image Generation using Residual Quantization](https://mlsanity.com/paper/2203.01941/) (2022)
- [AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks](https://mlsanity.com/paper/1711.10485/) (2017)
- [Generative Adversarial Networks: An Overview](https://mlsanity.com/paper/1710.07035/) (2017)
- [Cross-Modal Contrastive Learning for Text-to-Image Generation](https://mlsanity.com/paper/2101.04702/) (2021)
- [SnapFusion: Text-to-Image Diffusion Model on Mobile Devices within Two Seconds](https://mlsanity.com/paper/2306.00980/) (2023)
- [ReCo: Region-Controlled Text-to-Image Generation](https://mlsanity.com/paper/2211.15518/) (2022)
- [Depth-Aware Generative Adversarial Network for Talking Head Video Generation](https://mlsanity.com/paper/2203.06605/) (2022)
- [MirrorGAN: Learning Text-to-image Generation by Redescription](https://mlsanity.com/paper/1903.05854/) (2019)
- [DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesis](https://mlsanity.com/paper/2008.05865/) (2020)
- [SINE: SINgle Image Editing with Text-to-Image Diffusion Models](https://mlsanity.com/paper/2212.04489/) (2022)
- [StoryGAN: A Sequential Conditional GAN for Story Visualization](https://mlsanity.com/paper/1812.02784/) (2018)
- [Advancing Pose-Guided Image Synthesis with Progressive Conditional Diffusion Models](https://mlsanity.com/paper/2310.06313/) (2023)
- [Expressive Text-to-Image Generation with Rich Text](https://mlsanity.com/paper/2304.06720/) (2023)
- [V-Express: Conditional Dropout for Progressive Training of Portrait Video Generation](https://mlsanity.com/paper/2406.02511/) (2024)
- [Make-A-Story: Visual Memory Conditioned Consistent Story Generation](https://mlsanity.com/paper/2211.13319/) (2022)
- [StoryDALL-E: Adapting Pretrained Text-to-Image Transformers for Story Continuation](https://mlsanity.com/paper/2209.06192/) (2022)

## Built on by (most cited first, 30 of 54)

- [IMAGDressing-v1: Customizable Virtual Dressing](https://mlsanity.com/paper/2407.12705/) (2024)
- [Attentive Eraser: Unleashing Diffusion Model's Object Removal Potential via Self-Attention Redirection Guidance](https://mlsanity.com/paper/2412.12974/) (2024)
- [Efficient Diffusion Models: A Comprehensive Survey from Principles to Practices](https://mlsanity.com/paper/2410.11795/) (2024)
- [Generative AI for Cel-Animation: A Survey](https://mlsanity.com/paper/2501.06250/) (2025)
- [Story-Iter: A Training-free Iterative Paradigm for Long Story Visualization](https://mlsanity.com/paper/2410.06244/) (2024)
- [FaceShot: Bring Any Character into Life](https://mlsanity.com/paper/2503.00740/) (2025)
- [DVP-MVS: Synergize Depth-Edge and Visibility Prior for Multi-View Stereo](https://mlsanity.com/paper/2412.11578/) (2024)
- [Advanced Sign Language Video Generation with Compressed and Quantized Multi-Condition Tokenization](https://mlsanity.com/paper/2506.15980/) (2025)
- [ContextualStory: Consistent Visual Storytelling with Spatially-Enhanced and Storyline Context](https://mlsanity.com/paper/2407.09774/) (2024)
- [Causal Deciphering and Inpainting in Spatio-Temporal Dynamics via Diffusion Model](https://mlsanity.com/paper/2409.19608/) (2024)
- [Re-Attentional Controllable Video Diffusion Editing](https://mlsanity.com/paper/2412.11710/) (2024)
- [ViStoryBench: Comprehensive Benchmark Suite for Story Visualization](https://mlsanity.com/paper/2505.24862/) (2025)
- [ITVTON: Virtual Try-On Diffusion Transformer Based on Integrated Image and Text](https://mlsanity.com/paper/2501.16757/) (2025)
- [Dialogue Director: Bridging the Gap in Dialogue Visualization for Multimodal Storytelling](https://mlsanity.com/paper/2412.20725/) (2024)
- [Efficient Feature Fusion for UAV Object Detection](https://mlsanity.com/paper/2501.17983/) (2025)
- [Efficient Temporal Consistency in Diffusion-Based Video Editing with Adaptor Modules: A Theoretical Framework](https://mlsanity.com/paper/2504.16016/) (2025)
- [HiFiSeg: High-Frequency Information Enhanced Polyp Segmentation with Global-Local Vision Transformer](https://mlsanity.com/paper/2410.02528/) (2024)
- [Enhancing Multimodal Large Language Models Complex Reason via Similarity Computation](https://mlsanity.com/paper/2412.09817/) (2024)
- [AuxDepthNet: Real-Time Monocular 3D Object Detection with Depth-Sensitive Features](https://mlsanity.com/paper/2501.03700/) (2025)
- [Cross-modal Context Fusion and Adaptive Graph Convolutional Network for Multimodal Conversational Emotion Recognition](https://mlsanity.com/paper/2501.15063/) (2025)
- [HDM: Hybrid Diffusion Model for Unified Image Anomaly Detection](https://mlsanity.com/paper/2502.19200/) (2025)
- [An improved EfficientNetV2 for garbage classification](https://mlsanity.com/paper/2503.21208/) (2025)
- [CoCoIns: Consistent Subject Generation via Contrastive Instantiated Concepts](https://mlsanity.com/paper/2503.24387/) (2025)
- [MCFNet: A Multimodal Collaborative Fusion Network for Fine-Grained Semantic Classification](https://mlsanity.com/paper/2505.23365/) (2025)
- [MVL-Loc: Leveraging Vision-Language Model for Generalizable Multi-Scene Camera Relocalization](https://mlsanity.com/paper/2507.04509/) (2025)
- [ReCap: Lightweight Referential Grounding for Coherent Story Visualization](https://mlsanity.com/paper/2604.18575/) (2026)
- [Normality-Preserving Continual Industrial Anomaly Detection via Orthogonal LoRA Banks](https://mlsanity.com/paper/2606.02042/) (2026)
- [LSReGen: Large-Scale Regional Generator via Backward Guidance Framework](https://mlsanity.com/paper/2407.15066/) (2024)
- [DDNet: Deformable Convolution and Dense FPN for Surface Defect Detection in Recycled Books](https://mlsanity.com/paper/2409.04958/) (2024)
- [S3TU-Net: Structured Convolution and Superpixel Transformer for Lung Nodule Segmentation](https://mlsanity.com/paper/2411.12547/) (2024)

## Abstract (arXiv)

Recent research showcases the considerable potential of conditional diffusion models for generating consistent stories. However, current methods, which predominantly generate stories in an autoregressive and excessively caption-dependent manner, often underrate the contextual consistency and relevance of frames during sequential generation. To address this, we propose a novel Rich-contextual Conditional Diffusion Models (RCDMs), a two-stage approach designed to enhance story generation's semantic consistency and temporal consistency. Specifically, in the first stage, the frame-prior transformer diffusion model is presented to predict the frame semantic embedding of the unknown clip by aligning the semantic correlations between the captions and frames of the known clip. The second stage establishes a robust model with rich contextual conditions, including reference images of the known clip, the predicted frame semantic embedding of the unknown clip, and text embeddings of all captions. By jointly injecting these rich contextual conditions at the image and feature levels, RCDMs can generate semantic and temporal consistency stories. Moreover, RCDMs can generate consistent stories with a single forward inference compared to autoregressive models. Our qualitative and quantitative results demonstrate that our proposed RCDMs outperform in challenging scenarios. The code and model will be available at https://github.com/muzishen/RCDMs.