# Improved Visual Story Generation with Adaptive Context Modeling

arXiv: https://arxiv.org/abs/2305.16811 · HTML: https://mlsanity.com/paper/2305.16811/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 27 of its 36 references that are in the map.
- First posted on arXiv 26 May 2023.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers](https://mlsanity.com/paper/2211.01324/) (2022)
- [Generative Adversarial Text to Image Synthesis](https://mlsanity.com/paper/1605.05396/) (2016)
- [CogView: Mastering Text-to-Image Generation via Transformers](https://mlsanity.com/paper/2105.13290/) (2021)
- [Muse: Text-To-Image Generation via Masked Generative Transformers](https://mlsanity.com/paper/2301.00704/) (2023)
- [StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks](https://mlsanity.com/paper/1612.03242/) (2016)
- [Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors](https://mlsanity.com/paper/2203.13131/) (2022)
- [StoryGAN: A Sequential Conditional GAN for Story Visualization](https://mlsanity.com/paper/1812.02784/) (2018)
- [ERNIE-ViLG 2.0: Improving Text-to-Image Diffusion Model with Knowledge-Enhanced Mixture-of-Denoising-Experts](https://mlsanity.com/paper/2210.15257/) (2022)
- [StoryDALL-E: Adapting Pretrained Text-to-Image Transformers for Story Continuation](https://mlsanity.com/paper/2209.06192/) (2022)
- [Synthesizing Coherent Story with Auto-Regressive Latent Diffusion Models](https://mlsanity.com/paper/2211.10950/) (2022)
- [Imagine This! Scripts to Compositions to Videos](https://mlsanity.com/paper/1804.03608/) (2018)
- [Integrating Visuospatial, Linguistic and Commonsense Structure into Story Visualization](https://mlsanity.com/paper/2110.10834/) (2021)
- [Improving Generation and Evaluation of Visual Stories via Semantic Consistency](https://mlsanity.com/paper/2105.10026/) (2021)
- [UPainting: Unified Text-to-Image Diffusion Generation with Cross-modal Guidance](https://mlsanity.com/paper/2210.16031/) (2022)
- [Word-Level Fine-Grained Story Visualization](https://mlsanity.com/paper/2208.02341/) (2022)

## Built on by (most cited first, 22 of 22)

- [Training-Free Consistent Text-to-Image Generation](https://mlsanity.com/paper/2402.03286/) (2024)
- [MovieDreamer: Hierarchical Generation for Coherent Long Visual Sequence](https://mlsanity.com/paper/2407.16655/) (2024)
- [MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizer](https://mlsanity.com/paper/2401.10208/) (2024)
- [StoryWeaver: A Unified World Model for Knowledge-Enhanced Story Character Customization](https://mlsanity.com/paper/2412.07375/) (2024)
- [ContextualStory: Consistent Visual Storytelling with Spatially-Enhanced and Storyline Context](https://mlsanity.com/paper/2407.09774/) (2024)
- [Make-A-Storyboard: A General Framework for Storyboard with Disentangled and Merged Control](https://mlsanity.com/paper/2312.07549/) (2023)
- [Generating Coherent Sequences of Visual Illustrations for Real-World Manual Tasks](https://mlsanity.com/paper/2405.10122/) (2024)
- [RefDrop: Controllable Consistency in Image or Video Generation via Reference Feature Guidance](https://mlsanity.com/paper/2405.17661/) (2024)
- [ViStoryBench: Comprehensive Benchmark Suite for Story Visualization](https://mlsanity.com/paper/2505.24862/) (2025)
- [CookingDiffusion: Cooking Procedural Image Generation with Stable Diffusion](https://mlsanity.com/paper/2501.09042/) (2025)
- [Storybooth: Training-free Multi-Subject Consistency for Improved Visual Storytelling](https://mlsanity.com/paper/2504.05800/) (2025)
- [Contrastive Sequential-Diffusion Learning: Non-linear and Multi-Scene Instructional Video Synthesis](https://mlsanity.com/paper/2407.11814/) (2024)
- [Motion by Queries: Identity-Motion Trade-offs in Text-to-Video Generation](https://mlsanity.com/paper/2412.07750/) (2024)
- [ViSTA: Visual Storytelling using Multi-modal Adapters for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2506.12198/) (2025)
- [Masked Generative Story Transformer with Character Guidance and Caption Augmentation](https://mlsanity.com/paper/2403.08502/) (2024)
- [Improving Visual Storytelling with Multimodal Large Language Models](https://mlsanity.com/paper/2407.02586/) (2024)
- [StorySync: Training-Free Subject Consistency in Text-to-Image Generation via Region Harmonization](https://mlsanity.com/paper/2508.03735/) (2025)
- [LogiStory: A Logic-Aware Framework for Multi-Image Story Visualization](https://mlsanity.com/paper/2603.28082/) (2026)
- [FreeStory: Training-Free Character Consistency for Free-Form Visual Storytelling](https://mlsanity.com/paper/2606.25079/) (2026)
- [LLMs Behind the Scenes: Enabling Narrative Scene Illustration](https://mlsanity.com/paper/2509.22940/) (2025)
- [Adaptive Visual Conditioning for Semantic Consistency in Diffusion-Based Story Continuation](https://mlsanity.com/paper/2510.13787/) (2025)
- [Sidecar: Training-Free Semantic Reuse for Character-Consistent Free-form Visual Storytelling](https://mlsanity.com/paper/2608.27280/) (2026)

## Abstract (arXiv)

Diffusion models developed on top of powerful text-to-image generation models like Stable Diffusion achieve remarkable success in visual story generation. However, the best-performing approach considers historically generated results as flattened memory cells, ignoring the fact that not all preceding images contribute equally to the generation of the characters and scenes at the current stage. To address this, we present a simple method that improves the leading system with adaptive context modeling, which is not only incorporated in the encoder but also adopted as additional guidance in the sampling stage to boost the global consistency of the generated story. We evaluate our model on PororoSV and FlintstonesSV datasets and show that our approach achieves state-of-the-art FID scores on both story visualization and continuation scenarios. We conduct detailed model analysis and show that our model excels at generating semantically consistent images for stories.