mlsanityData to 7 Oct 2026

Instruct-Imagen: Image Generation with Multi-modal Instruction

arXiv 2401.01952 ↗First posted 3 Jan 2024cs.CV cs.AI cs.CLcited by 65

Not on the map. Why?

Abstract (arXiv)

This paper presents instruct-imagen, a model that tackles heterogeneous image generation tasks and generalizes across unseen tasks. We introduce *multi-modal instruction* for image generation, a task representation articulating a range of generation intents with precision. It uses natural language to amalgamate disparate modalities (e.g., text, edge, style, subject, etc.), such that abundant generation intents can be standardized in a uniform format. We then build instruct-imagen by fine-tuning a pre-trained text-to-image diffusion model with a two-stage framework. First, we adapt the model using the retrieval-augmented training, to enhance model's capabilities to ground its generation on external multimodal context. Subsequently, we fine-tune the adapted model on diverse image generation tasks that requires vision-language understanding (e.g., subject-driven generation, etc.), each paired with a multi-modal instruction encapsulating the task's essence. Human evaluation on various image generation datasets reveals that instruct-imagen matches or surpasses prior task-specific models in-domain and demonstrates promising generalization to unseen and more complex tasks.

Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). Builds on 47 of its 54 references that are in the map. First posted on arXiv 3 Jan 2024.

Builds on

47 of 54 references.

References · 47

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  12. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
35 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  3. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  4. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  6. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  8. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  9. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  10. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  11. 2023PaLM 2 Technical Reportcited by 1,154
  12. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  13. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  14. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  15. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  16. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  17. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  18. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  19. 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 615
  20. 2015Holistically-Nested Edge Detectioncited by 590
  21. 2023BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editingcited by 438
  22. 2017NIMA: Neural Image Assessmentcited by 387
  23. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  24. 2023Subject-driven Text-to-Image Generation via Apprenticeship Learningcited by 210
  25. 2022Re-Imagen: Retrieval-Augmented Text-to-Image Generatorcited by 208
  26. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  27. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 203
  28. 2023StyleDrop: Text-to-Image Generation in Any Stylecited by 197
  29. 2023Taming Encoder for Zero Fine-tuning Image Customization with Text-to-Image Diffusion Modelscited by 116
  30. 2023Open-domain Visual Entity Recognition: Towards Recognizing Millions of Wikipedia Entitiescited by 112
  31. 2023Kosmos-G: Generating Images in Context with Multimodal Large Language Modelscited by 105
  32. 2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Usecited by 100
  33. 2017Exploring the structure of a real-time, arbitrary neural artistic stylization networkcited by 89
  34. 2023ImagenHub: Standardizing the evaluation of conditional image generation modelscited by 75
  35. 2019Photo-Sketching: Inferring Contour Drawings from Imagescited by 67

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 27; 2025: 25; 2026: 13.

65 papers.

Sort:

By area:

  1. 2025Step1X-Edit: A Practical Framework for General Image Editingcited by 417
  2. 2025Transfer between Modalities with MetaQueriescited by 231
  3. 2024MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidancecited by 142
  4. 2024DreamBench++: A Human-Aligned Benchmark for Personalized Image Generationcited by 131
  5. 2024UniReal: Universal Image Generation and Editing via Learning Real-world Dynamicscited by 100
  6. 2025UniVideo: Unified Understanding, Generation, and Editing for Videoscited by 85
  7. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  8. 2024Diffusion Feedback Helps CLIP See Bettercited by 37
  9. 2025CreatiDesign: A Unified Multi-Conditional Diffusion Transformer for Creative Graphic Designcited by 36
  10. 2024Reward Guided Latent Consistency Distillationcited by 35
  11. 2024MusicMagus: Zero-Shot Text-to-Music Editing via Diffusion Modelscited by 32
  12. 2024PixWizard: Versatile Image-to-Image Visual Assistant with Open-Language Instructionscited by 31
38 more
  1. 2024Enhance Image-to-Image Generation with LLaVA-generated Promptscited by 20
  2. 2025ImageRAG: Dynamic Image Retrieval for Reference-Guided Image Generationcited by 20
  3. 2025Personalized Image Generation with Deep Generative Models: A Decade Surveycited by 20
  4. 2024RealCompo: Balancing Realism and Compositionality Improves Text-to-Image Diffusion Modelscited by 18
  5. 2024ObjectMate: A Recurrence Prior for Object Insertion and Subject-Driven Generationcited by 16
  6. 2025Mod-Adapter: Tuning-Free and Versatile Multi-concept Personalization via Modulation Adaptercited by 13
  7. 2024PrimeDepth: Efficient Monocular Depth Estimation with a Stable Diffusion Preimagecited by 11
  8. 2024AttnDreamBooth: Towards Text-Aligned Personalized Text-to-Image Generationcited by 10
  9. 2025MIGE: Mutually Enhanced Multimodal Instruction-Based Image Generation and Editingcited by 10
  10. 2024Modality-Specialized Synergizers for Interleaved Vision-Language Generalistscited by 9
  11. 2024UNIC-Adapter: Unified Image-instruction Adapter with Multi-modal Transformer for Image Generationcited by 9
  12. 2025MultiRef: Controllable Image Generation with Multiple Visual Referencescited by 9
  13. 2024VIMI: Grounding Video Generation through Multi-modal Instructioncited by 8
  14. 2024DreamStruct: Understanding Slides and User Interfaces via Synthetic Data Generationcited by 8
  15. 2025Visual Persona: Foundation Model for Full-Body Human Customizationcited by 7
  16. 2025BlenderFusion: 3D-Grounded Visual Editing and Generative Compositingcited by 7
  17. 2025ShoulderShot: Generating Over-the-Shoulder Dialogue Videoscited by 7
  18. 2024Improving Subject-Driven Image Synthesis with Subject-Agnostic Guidancecited by 6
  19. 2025Multi-party Collaborative Attention Control for Image Customizationcited by 6
  20. 2025CDST: Color Disentangled Style Transfer for Universal Style Reference Customizationcited by 6
  21. 2026ImagenWorld: Stress-Testing Image Generation Models with Explainable Human Evaluation on Open-ended Real-World Taskscited by 6
  22. 2026Search Beyond What Can Be Taught: Evolving the Knowledge Boundary in Agentic Visual Generationcited by 6
  23. 2024Towards General Text-guided Image Synthesis for Customized Multimodal Brain MRI Generationcited by 5
  24. 2024Style-Friendly SNR Sampler for Style-Driven Generationcited by 5
  25. 2025[De|Re]constructing VLMs' Reasoning in Countingcited by 5
  26. 2024MUMU: Bootstrapping Multimodal Image Generation from Text-to-Image Datacited by 4
  27. 2024KITTEN: A Knowledge-Intensive Evaluation of Image Generation on Visual Entitiescited by 4
  28. 2025ImageSentinel: Protecting Visual Datasets from Unauthorized Retrieval-Augmented Image Generationcited by 4
  29. 2025EMMA: Concept Erasure Benchmark with Comprehensive Semantic Metrics and Diverse Categoriescited by 4
  30. 2024Openstory++: A Large-scale Dataset and Benchmark for Instance-aware Open-domain Visual Storytellingcited by 3
  31. 2025VoCap: Video Object Captioning and Segmentation from Any Promptcited by 3
  32. 2026DSH-Bench: A Difficulty- and Scenario-Aware Benchmark with Hierarchical Subject Taxonomy for Subject-Driven Text-to-Image Generationcited by 3
  33. 2026CREval: An Automated Interpretable Evaluation for Creative Image Manipulation under Complex Instructionscited by 3
  34. 2026Pareto LoRA: Mitigating Modality Imbalance in Unified Multimodal Models via Pareto-Optimal Gradient Integrationcited by 3
  35. 2024Denoising Monte Carlo Renders with Diffusion Modelscited by 2
  36. 2024MasterWeaver: Taming Editability and Face Identity for Personalized Text-to-Image Generationcited by 2
  37. 2024AnyLogo: Symbiotic Subject-Driven Diffusion System with Gemini Statuscited by 2
  38. 2025COP-GEN-Beta: Unified Generative Modelling of COPernicus Imagery Thumbnailscited by 2

50 most cited of 65 All 65 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalists
  2. 2023InstructAny2Pix: Flexible Visual Editing via Multimodal Instruction Following
  3. 2026Images in Sentences: Scaling Interleaved Instructions for Unified Visual Generation
  4. 2023VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluation
  5. 2022Write and Paint: Generative Vision-Language Models are Unified Modal Learners
  6. 2024Multi-modal Generation via Cross-Modal In-Context Learning
  7. 2025UniModel: A Visual-Only Framework for Unified Multimodal Understanding and Generation
  8. 2023MiniGPT-5: Interleaved Vision-and-Language Generation via Generative Vokens