mlsanityData to 7 Oct 2026

Generating Illustrated Instructions

arXiv 2312.04552 ↗First posted 7 Dec 2023cs.CV cs.AI cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

We introduce the new task of generating Illustrated Instructions, i.e., visual instructions customized to a user's needs. We identify desiderata unique to this task, and formalize it through a suite of automatic and human evaluation metrics, designed to measure the validity, consistency, and efficacy of the generations. We combine the power of large language models (LLMs) together with strong text-to-image generation diffusion models to propose a simple approach called StackedDiffusion, which generates such illustrated instructions given text as input. The resulting model strongly outperforms baseline approaches and state-of-the-art multimodal LLMs; and in 30% of cases, users even prefer it to human-generated articles. Most notably, it enables various new and exciting applications far beyond what static articles on the web can provide, such as personalized instructions complete with intermediate steps and pictures in response to a user's individual situation.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 67 of its 80 references that are in the map. First posted on arXiv 7 Dec 2023.

Builds on

67 of 80 references.

References · 67

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  8. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  9. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  11. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  12. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
38 more
  1. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  2. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  3. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  4. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  5. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  6. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  7. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  8. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  9. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  10. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  11. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  12. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 589
  13. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  14. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  15. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 463
  16. 2020Multi-modal Transformer for Video Retrievalcited by 395
  17. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  18. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generationcited by 330
  19. 2019COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysiscited by 316
  20. 2023Common Diffusion Noise Schedules and Sample Steps are Flawedcited by 308
  21. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  22. 2023Generating Images with Multimodal Language Modelscited by 259
  23. 2020UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generationcited by 258
  24. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 254
  25. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  26. 2018WikiHow: A Large Scale Text Summarization Datasetcited by 203
  27. 2023Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion modelscited by 199
  28. 2021Anticipative Video Transformercited by 197
  29. 2022MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognitioncited by 189
  30. 2023NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generationcited by 170
  31. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuningcited by 168
  32. 2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142
  33. 2023Grounding Language Models to Images for Multimodal Inputs and Outputscited by 129
  34. 2020Rolling-Unrolling LSTMs for Action Anticipation from First-Person Videocited by 119
  35. 2021TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignmentcited by 102
  36. 2021CLIP-It! Language-Guided Video Summarizationcited by 99
  37. 2021Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformerscited by 93
  38. 2021HiT: Hierarchical Transformer with Momentum Contrast for Video-Text Retrievalcited by 87

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 4; 2026: 1.

7 papers.

  1. 2024ShowHowTo: Generating Scene-Conditioned Step-by-Step Visual Instructionscited by 7
  2. 2025Long-horizon Visual Instruction Generation with Logic and Attribute Self-reflectioncited by 4
  3. 2024Coherent Zero-Shot Visual Instruction Generationcited by 2
  4. 2025Stitch-a-Demo: Video Demonstrations from Multistep Descriptionscited by 1
  5. 2025Latent Beam Diffusion Models for Generating Visual Sequencescited by 1
  6. 2026Order Matters: LVLMs as Judges for Temporal Reasoning in Image Sequencescited by 1
  7. 2025Show Me: Unifying Instructional Image and Video Generation with Diffusion Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025DesignDiffusion: High-Quality Text-to-Design Image Generation with Diffusion Models
  2. 2023LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts
  3. 2025Draw ALL Your Imagine: A Holistic Benchmark and Agent Framework for Complex Instruction-based Image Generation
  4. 2023Reason out Your Layout: Evoking the Layout Master from Large Language Models for Text-to-Image Synthesis
  5. 2025From Real to Synthetic: Synthesizing Millions of Diversified and Complicated User Instructions with Attributed Grounding
  6. 2024Generating Coherent Sequences of Visual Illustrations for Real-World Manual Tasks
  7. 2023LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models
  8. 2025Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis