# Generating Coherent Sequences of Visual Illustrations for Real-World Manual Tasks

arXiv: https://arxiv.org/abs/2405.10122 · HTML: https://mlsanity.com/paper/2405.10122/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 35 references that are in the map.
- First posted on arXiv 16 May 2024.

## Builds on (most cited first, 20 of 20)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Visual Storytelling](https://mlsanity.com/paper/1604.03968/) (2016)
- [StoryGAN: A Sequential Conditional GAN for Story Visualization](https://mlsanity.com/paper/1812.02784/) (2018)
- [Make-A-Story: Visual Memory Conditioned Consistent Story Generation](https://mlsanity.com/paper/2211.13319/) (2022)
- [StoryDALL-E: Adapting Pretrained Text-to-Image Transformers for Story Continuation](https://mlsanity.com/paper/2209.06192/) (2022)
- [Synthesizing Coherent Story with Auto-Regressive Latent Diffusion Models](https://mlsanity.com/paper/2211.10950/) (2022)
- [DeepStory: Video Story QA by Deep Embedded Memory Networks](https://mlsanity.com/paper/1707.00836/) (2017)
- [Guided Image Synthesis via Initial Image Editing in Diffusion Model](https://mlsanity.com/paper/2305.03382/) (2023)
- [Imagine This! Scripts to Compositions to Videos](https://mlsanity.com/paper/1804.03608/) (2018)
- [Multimodal Procedural Planning via Dual Text-Image Prompting](https://mlsanity.com/paper/2305.01795/) (2023)
- [aMUSEd: An Open MUSE Reproduction](https://mlsanity.com/paper/2401.01808/) (2024)
- [Improved Visual Story Generation with Adaptive Context Modeling](https://mlsanity.com/paper/2305.16811/) (2023)
- [Plot and Rework: Modeling Storylines for Visual Storytelling](https://mlsanity.com/paper/2105.06950/) (2021)
- [A Benchmark of Visual Storytelling in Social Media](https://mlsanity.com/paper/1908.03505/) (2019)
- [Assisting News Media Editors with Cohesive Visual Storylines](https://mlsanity.com/paper/2110.06805/) (2021)

## Built on by (most cited first, 6 of 6)

- [Visual Riddles: a Commonsense and World Knowledge Challenge for Large Vision and Language Models](https://mlsanity.com/paper/2407.19474/) (2024)
- [ShowHowTo: Generating Scene-Conditioned Step-by-Step Visual Instructions](https://mlsanity.com/paper/2412.01987/) (2024)
- [Long-horizon Visual Instruction Generation with Logic and Attribute Self-reflection](https://mlsanity.com/paper/2503.13500/) (2025)
- [Coherent Zero-Shot Visual Instruction Generation](https://mlsanity.com/paper/2406.04337/) (2024)
- [Contrastive Sequential-Diffusion Learning: Non-linear and Multi-Scene Instructional Video Synthesis](https://mlsanity.com/paper/2407.11814/) (2024)
- [Latent Beam Diffusion Models for Generating Visual Sequences](https://mlsanity.com/paper/2503.20429/) (2025)

## Abstract (arXiv)

Multistep instructions, such as recipes and how-to guides, greatly benefit from visual aids, such as a series of images that accompany the instruction steps. While Large Language Models (LLMs) have become adept at generating coherent textual steps, Large Vision/Language Models (LVLMs) are less capable of generating accompanying image sequences. The most challenging aspect is that each generated image needs to adhere to the relevant textual step instruction, as well as be visually consistent with earlier images in the sequence. To address this problem, we propose an approach for generating consistent image sequences, which integrates a Latent Diffusion Model (LDM) with an LLM to transform the sequence into a caption to maintain the semantic coherence of the sequence. In addition, to maintain the visual coherence of the image sequence, we introduce a copy mechanism to initialise reverse diffusion processes with a latent vector iteration from a previously generated image from a relevant step. Both strategies will condition the reverse diffusion process on the sequence of instruction steps and tie the contents of the current image to previous instruction steps and corresponding images. Experiments show that the proposed approach is preferred by humans in 46.6% of the cases against 26.6% for the second best method. In addition, automatic metrics showed that the proposed method maintains semantic coherence and visual consistency across steps in both domains.