# Character-Centric Story Visualization via Visual Planning and Token Alignment

arXiv: https://arxiv.org/abs/2210.08465 · HTML: https://mlsanity.com/paper/2210.08465/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 37 references that are in the map.
- First posted on arXiv 16 Oct 2022.

## Builds on (most cited first, 24 of 24)

- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks](https://mlsanity.com/paper/1710.11063/) (2017)
- [CogView: Mastering Text-to-Image Generation via Transformers](https://mlsanity.com/paper/2105.13290/) (2021)
- [AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks](https://mlsanity.com/paper/1711.10485/) (2017)
- [VideoGPT: Video Generation using VQ-VAE and Transformers](https://mlsanity.com/paper/2104.10157/) (2021)
- [Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networks](https://mlsanity.com/paper/1910.01279/) (2019)
- [StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks](https://mlsanity.com/paper/1612.03242/) (2016)
- [A Semantic Loss Function for Deep Learning with Symbolic Knowledge](https://mlsanity.com/paper/1711.11157/) (2017)
- [Plan-And-Write: Towards Better Automatic Storytelling](https://mlsanity.com/paper/1811.05701/) (2018)
- [Eigen-CAM: Class Activation Map using Principal Components](https://mlsanity.com/paper/2008.00299/) (2020)
- [MirrorGAN: Learning Text-to-image Generation by Redescription](https://mlsanity.com/paper/1903.05854/) (2019)
- [Axiom-based Grad-CAM: Towards Accurate Visualization and Explanation of CNNs](https://mlsanity.com/paper/2008.02312/) (2020)
- [StoryGAN: A Sequential Conditional GAN for Story Visualization](https://mlsanity.com/paper/1812.02784/) (2018)
- [Content Planning for Neural Story Generation with Aristotelian Rescoring](https://mlsanity.com/paper/2009.09870/) (2020)
- [Imagine This! Scripts to Compositions to Videos](https://mlsanity.com/paper/1804.03608/) (2018)
- [Integrating Visuospatial, Linguistic and Commonsense Structure into Story Visualization](https://mlsanity.com/paper/2110.10834/) (2021)
- [Plan, Write, and Revise: an Interactive System for Open-Domain Story Generation](https://mlsanity.com/paper/1904.02357/) (2019)
- [Improving Generation and Evaluation of Visual Stories via Semantic Consistency](https://mlsanity.com/paper/2105.10026/) (2021)
- [Go Back in Time: Generating Flashbacks in Stories with Event Temporal Prompts](https://mlsanity.com/paper/2205.01898/) (2022)

## Built on by (most cited first, 30 of 33)

- [Intelligent Grimm -- Open-ended Visual Storytelling via Latent Diffusion Models](https://mlsanity.com/paper/2306.00973/) (2023)
- [Make-A-Story: Visual Memory Conditioned Consistent Story Generation](https://mlsanity.com/paper/2211.13319/) (2022)
- [TaleCrafter: Interactive Story Visualization with Multiple Characters](https://mlsanity.com/paper/2305.18247/) (2023)
- [Synthesizing Coherent Story with Auto-Regressive Latent Diffusion Models](https://mlsanity.com/paper/2211.10950/) (2022)
- [Boosting Consistency in Story Visualization with Rich-Contextual Conditional Diffusion Models](https://mlsanity.com/paper/2407.02482/) (2024)
- [DreamStory: Open-Domain Story Visualization by LLM-Guided Multi-Subject Consistent Diffusion](https://mlsanity.com/paper/2407.12899/) (2024)
- [AutoStory: Generating Diverse Storytelling Images with Minimal Human Effort](https://mlsanity.com/paper/2311.11243/) (2023)
- [Anim-Director: A Large Multimodal Model Powered Agent for Controllable Animation Video Generation](https://mlsanity.com/paper/2408.09787/) (2024)
- [StoryAgent: Customized Storytelling Video Generation via Multi-Agent Collaboration](https://mlsanity.com/paper/2411.04925/) (2024)
- [StoryGPT-V: Large Language Models as Consistent Story Visualizers](https://mlsanity.com/paper/2312.02252/) (2023)
- [StoryImager: A Unified and Efficient Framework for Coherent Story Visualization and Completion](https://mlsanity.com/paper/2404.05979/) (2024)
- [AesopAgent: Agent-driven Evolutionary System on Story-to-Video Production](https://mlsanity.com/paper/2403.07952/) (2024)
- [Story-Iter: A Training-free Iterative Paradigm for Long Story Visualization](https://mlsanity.com/paper/2410.06244/) (2024)
- [A survey on knowledge-enhanced multimodal learning](https://mlsanity.com/paper/2211.12328/) (2022)
- [Causal-Story: Local Causal Attention Utilizing Parameter-Efficient Tuning For Visual Story Synthesis](https://mlsanity.com/paper/2309.09553/) (2023)
- [ContextualStory: Consistent Visual Storytelling with Spatially-Enhanced and Storyline Context](https://mlsanity.com/paper/2407.09774/) (2024)
- [One missing piece in Vision and Language: A Survey on Comics Understanding](https://mlsanity.com/paper/2409.09502/) (2024)
- [Story Visualization by Online Text Augmentation with Context Memory](https://mlsanity.com/paper/2308.07575/) (2023)
- [Make-A-Storyboard: A General Framework for Storyboard with Disentangled and Merged Control](https://mlsanity.com/paper/2312.07549/) (2023)
- [Object Isolated Attention for Consistent Story Visualization](https://mlsanity.com/paper/2503.23353/) (2025)
- [ViStoryBench: Comprehensive Benchmark Suite for Story Visualization](https://mlsanity.com/paper/2505.24862/) (2025)
- [KAHANI: Culturally-Nuanced Visual Storytelling Tool for Non-Western Cultures](https://mlsanity.com/paper/2410.19419/) (2024)
- [VinaBench: Benchmark for Faithful and Consistent Visual Narratives](https://mlsanity.com/paper/2503.20871/) (2025)
- [FairyGen: Storied Cartoon Video from a Single Child-Drawn Character](https://mlsanity.com/paper/2506.21272/) (2025)
- [Improving Multi-Subject Consistency in Open-Domain Image Generation with Isolation and Reposition Attention](https://mlsanity.com/paper/2411.19261/) (2024)
- [StoryReasoning Dataset: Using Chain-of-Thought for Scene Understanding and Grounded Story Generation](https://mlsanity.com/paper/2505.10292/) (2025)
- [Masked Generative Story Transformer with Character Guidance and Caption Augmentation](https://mlsanity.com/paper/2403.08502/) (2024)
- [REFFLY: Melody-Constrained Lyrics Editing Model](https://mlsanity.com/paper/2409.00292/) (2024)
- [ReCap: Lightweight Referential Grounding for Coherent Story Visualization](https://mlsanity.com/paper/2604.18575/) (2026)
- [MangaFlow: An End-to-End Agentic Framework for Controllable Story to Manga Generation](https://mlsanity.com/paper/2605.28173/) (2026)

## Abstract (arXiv)

Story visualization advances the traditional text-to-image generation by enabling multiple image generation based on a complete story. This task requires machines to 1) understand long text inputs and 2) produce a globally consistent image sequence that illustrates the contents of the story. A key challenge of consistent story visualization is to preserve characters that are essential in stories. To tackle the challenge, we propose to adapt a recent work that augments Vector-Quantized Variational Autoencoders (VQ-VAE) with a text-tovisual-token (transformer) architecture. Specifically, we modify the text-to-visual-token module with a two-stage framework: 1) character token planning model that predicts the visual tokens for characters only; 2) visual token completion model that generates the remaining visual token sequence, which is sent to VQ-VAE for finalizing image generations. To encourage characters to appear in the images, we further train the two-stage framework with a character-token alignment objective. Extensive experiments and evaluations demonstrate that the proposed method excels at preserving characters and can produce higher quality image sequences compared with the strong baselines. Codes can be found in https://github.com/sairin1202/VP-CSV