# InstructionCrafter: Generating Consistent and High-Fidelity Visual Instructions

arXiv: https://arxiv.org/abs/2608.08460 · HTML: https://mlsanity.com/paper/2608.08460/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 148 references that are in the map.
- First posted on arXiv 9 Aug 2026.

## Builds on (most cited first, 27 of 27)

- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023)
- [DINOv3](https://mlsanity.com/paper/2508.10104/) (2025)
- [FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space](https://mlsanity.com/paper/2506.15742/) (2025)
- [Open-Sora: Democratizing Efficient Video Production for All](https://mlsanity.com/paper/2412.20404/) (2024)
- [Latte: Latent Diffusion Transformer for Video Generation](https://mlsanity.com/paper/2401.03048/) (2024)
- [VideoCrafter1: Open Diffusion Models for High-Quality Video Generation](https://mlsanity.com/paper/2310.19512/) (2023)
- [Video models are zero-shot learners and reasoners](https://mlsanity.com/paper/2509.20328/) (2025)
- [MakeAnything: Harnessing Diffusion Transformers for Multi-Domain Procedural Sequence Generation](https://mlsanity.com/paper/2502.01572/) (2025)
- [Visual Goal-Step Inference using wikiHow](https://mlsanity.com/paper/2104.05845/) (2021)
- [StoryMaker: Towards Holistic Consistent Characters in Text-to-image Generation](https://mlsanity.com/paper/2409.12576/) (2024)
- [Contrastive Test-Time Composition of Multiple LoRA Models for Image Generation](https://mlsanity.com/paper/2403.19776/) (2024)
- [Coherent Zero-Shot Visual Instruction Generation](https://mlsanity.com/paper/2406.04337/) (2024)
- [Consistent Story Generation: Unlocking the Potential of Zigzag Sampling](https://mlsanity.com/paper/2506.09612/) (2025)
- [DuoGen: Towards General Purpose Interleaved Multimodal Generation](https://mlsanity.com/paper/2602.00508/) (2026)
- [StorySync: Training-Free Subject Consistency in Text-to-Image Generation via Region Harmonization](https://mlsanity.com/paper/2508.03735/) (2025)
- [I2G: Generating Instructional Illustrations via Text-Conditioned Diffusion](https://mlsanity.com/paper/2505.16425/) (2025)

## Abstract (arXiv)

Given textual task instructions, generating step-by-step visual instructions as an image sequence requires the simultaneous satisfaction of multiple properties, specifically step faithfulness, cross-image consistency, and per-frame visual quality. Existing text-to-image generation approaches rarely meet all three properties, owing to independent sampling that breaks consistency, finetuning on low-quality video that degrades per-frame quality, and frozen backbones that lack multi-step understanding. In this work, we propose InstructionCrafter, a diffusion-based framework with the key idea of separating the optimization of temporal and instructional alignment from per-frame visual quality via (1) spatial-freeze training and (2) instruction-aware adapters. Built on a pretrained video diffusion backbone, InstructionCrafter freezes the spatial layers that control per-frame detail and updates only temporal and text-conditioning pathways to learn instruction semantics and inter-step relations, which preserves the generative prior for per-frame quality and reduces trainable parameters by about 50 percent compared with full finetuning. We also introduce two lightweight adapters that enhance the model's understanding of instructional context. The Consistent Adapter aggregates textual cues from the entire instruction sequence and from neighboring steps to keep object identity and attributes consistent across frames, and the Context-Aware Temporal Adapter converts cross-attention outputs into biases for temporal self-attention, explicitly propagating inter-frame relations. Extensive experiments on two benchmark datasets demonstrate state-of-the-art overall performance on step faithfulness, cross-image consistency, and per-frame visual quality while significantly reducing noise, blur, and spurious subtitles. Our code and trained models will be publicly available.