# ViSTA: Visual Storytelling using Multi-modal Adapters for Text-to-Image Diffusion Models

arXiv: https://arxiv.org/abs/2506.12198 · HTML: https://mlsanity.com/paper/2506.12198/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 36 of its 46 references that are in the map.
- First posted on arXiv 13 Jun 2025.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 36)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [UnifiedQA: Crossing Format Boundaries With a Single QA System](https://mlsanity.com/paper/2005.00700/) (2020)
- [TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering](https://mlsanity.com/paper/2303.11897/) (2023)
- [StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation](https://mlsanity.com/paper/2405.01434/) (2024)
- [Training-Free Consistent Text-to-Image Generation](https://mlsanity.com/paper/2402.03286/) (2024)
- [MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizer](https://mlsanity.com/paper/2401.10208/) (2024)
- [Intelligent Grimm -- Open-ended Visual Storytelling via Latent Diffusion Models](https://mlsanity.com/paper/2306.00973/) (2023)
- [Make-A-Story: Visual Memory Conditioned Consistent Story Generation](https://mlsanity.com/paper/2211.13319/) (2022)
- [TaleCrafter: Interactive Story Visualization with Multiple Characters](https://mlsanity.com/paper/2305.18247/) (2023)
- [SEED-Story: Multimodal Long Story Generation with Large Language Model](https://mlsanity.com/paper/2407.08683/) (2024)
- [Synthesizing Coherent Story with Auto-Regressive Latent Diffusion Models](https://mlsanity.com/paper/2211.10950/) (2022)
- [The Chosen One: Consistent Characters in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2311.10093/) (2023)
- [Integrating Visuospatial, Linguistic and Commonsense Structure into Story Visualization](https://mlsanity.com/paper/2110.10834/) (2021)
- [One-Prompt-One-Story: Free-Lunch Consistent Text-to-Image Generation Using a Single Prompt](https://mlsanity.com/paper/2501.13554/) (2025)
- [DreamStory: Open-Domain Story Visualization by LLM-Guided Multi-Subject Consistent Diffusion](https://mlsanity.com/paper/2407.12899/) (2024)
- [StoryMaker: Towards Holistic Consistent Characters in Text-to-image Generation](https://mlsanity.com/paper/2409.12576/) (2024)
- [Zero-shot Generation of Coherent Storybook from Plain Text Story using Diffusion Models](https://mlsanity.com/paper/2302.03900/) (2023)
- [StoryGPT-V: Large Language Models as Consistent Story Visualizers](https://mlsanity.com/paper/2312.02252/) (2023)

## Built on by (most cited first, 3 of 3)

- [TRACE: Task-Adaptive Reasoning and Representation Learning for Universal Multimodal Retrieval](https://mlsanity.com/paper/2603.02929/) (2026)
- [FreeStory: Training-Free Character Consistency for Free-Form Visual Storytelling](https://mlsanity.com/paper/2606.25079/) (2026)
- [Sidecar: Training-Free Semantic Reuse for Character-Consistent Free-form Visual Storytelling](https://mlsanity.com/paper/2608.27280/) (2026)

## Abstract (arXiv)

Text-to-image diffusion models have achieved remarkable success, yet generating coherent image sequences for visual storytelling remains challenging. A key challenge is effectively leveraging all previous text-image pairs, referred to as history text-image pairs, which provide contextual information for maintaining consistency across frames. Existing auto-regressive methods condition on all past image-text pairs but require extensive training, while training-free subject-specific approaches ensure consistency but lack adaptability to narrative prompts. To address these limitations, we propose a multi-modal history adapter for text-to-image diffusion models, ViSTA. It consists of (1) a multi-modal history fusion module to extract relevant history features and (2) a history adapter to condition the generation on the extracted relevant features. We also introduce a salient history selection strategy during inference, where the most salient history text-image pair is selected, improving the quality of the conditioning. Furthermore, we propose to employ a Visual Question Answering-based metric TIFA to assess text-image alignment in visual storytelling, providing a more targeted and interpretable assessment of generated images. Evaluated on the StorySalon and FlintStonesSV dataset, our proposed ViSTA model is not only consistent across different frames, but also well-aligned with the narrative text descriptions.