mlsanityData to 14 Oct 2026

InstructDiffusion: A Generalist Modeling Interface for Vision Tasks

arXiv 2309.03895 ↗First posted 7 Sep 2023cs.CVcited by 164

Not on the map. Why?

Cited by 65 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present InstructDiffusion, a unifying and generic framework for aligning computer vision tasks with human instructions. Unlike existing approaches that integrate prior knowledge and pre-define the output space (e.g., categories and coordinates) for each vision task, we cast diverse vision tasks into a human-intuitive image-manipulating process whose output space is a flexible and interactive pixel space. Concretely, the model is built upon the diffusion process and is trained to predict pixels according to user instructions, such as encircling the man's left shoulder in red or applying a blue mask to the left car. InstructDiffusion could handle a variety of vision tasks, including understanding tasks (such as segmentation and keypoint detection) and generative tasks (such as editing and enhancement). It even exhibits the ability to handle unseen tasks and outperforms prior methods on novel datasets. This represents a significant step towards a generalist modeling interface for vision tasks, advancing artificial general intelligence in the field of computer vision.

Key facts. Cited by 164 papers in the map (data through 14 Oct 2026). Builds on 89 of its 103 references that are in the map. First posted on arXiv 7 Sep 2023.

Builds on

89 of 103 references.

References · 89

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  7. 2023GPT-4 Technical Reportcited by 18,396
  8. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  9. 2022Training language models to follow instructions with human feedbackcited by 15,971
  10. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  11. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  12. 2023Segment Anythingcited by 8,234
38 more
  1. 2023Visual Instruction Tuningcited by 7,869
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  3. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  4. 2022Classifier-Free Diffusion Guidancecited by 5,847
  5. 2022Generative Adversarial Networkscited by 5,627
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2017Neural Discrete Representation Learningcited by 4,712
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  13. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  14. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  15. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  16. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  17. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  18. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,387
  19. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  20. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  21. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,179
  22. 2023PaLM 2 Technical Reportcited by 1,162
  23. 2021Grounded Language-Image Pre-trainingcited by 1,156
  24. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  25. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  26. 2016Modeling Context in Referring Expressionscited by 1,063
  27. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  28. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  29. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 864
  30. 2022A Generalist Agentcited by 808
  31. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  32. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 774
  33. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  34. 2021Resolution-robust Large Mask Inpainting with Fourier Convolutionscited by 738
  35. 2016Deep Multi-scale Convolutional Neural Network for Dynamic Scene Deblurringcited by 727
  36. 2022Simple Baselines for Image Restorationcited by 708
  37. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  38. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649

50 most cited of 89 All 89 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 14; 2024: 59; 2025: 64; 2026: 27.

164 papers.

Sort:

By area:

  1. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 478
  2. 2024OmniGen: Unified Image Generationcited by 394
  3. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  4. 2025ImgEdit: A Unified Image Editing Dataset and Benchmarkcited by 231
  5. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  6. 2023SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Modelscited by 177
  7. 2023Direct Inversion: Boosting Diffusion-based Editing with 3 Lines of Codecited by 145
  8. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 137
  9. 2024GenArtist: Multimodal LLM as an Agent for Unified Image Generation and Editingcited by 133
  10. 2025X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Againcited by 112
  11. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Modelscited by 94
  12. 2024I2EBench: A Comprehensive Benchmark for Instruction-based Image Editingcited by 63
38 more
  1. 2024Attentive Eraser: Unleashing Diffusion Model's Object Removal Potential via Self-Attention Redirection Guidancecited by 63
  2. 2023Focus on Your Instruction: Fine-grained and Multi-instruction Image Editing by Attention Modulationcited by 62
  3. 2025Se~norita-2M: A High-Quality Instruction-based Dataset for General Video Editing by Video Specialistscited by 61
  4. 2024ACE: All-round Creator and Editor Following Instructions via Diffusion Transformercited by 56
  5. 2024Diffusion Models and Representation Learning: A Surveycited by 54
  6. 2024MineDreamer: Learning to Follow Instructions via Chain-of-Imagination for Simulated-World Controlcited by 53
  7. 2025MiniMax-Remover: Taming Bad Noise Helps Video Object Removalcited by 45
  8. 2025Reflect-DiT: Inference-Time Scaling for Text-to-Image Diffusion Transformers via In-Context Reflectioncited by 43
  9. 2025Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decodingcited by 43
  10. 2024One Diffusion to Generate Them Allcited by 41
  11. 2024PromptFix: You Prompt and We Fix the Photocited by 40
  12. 2024Improved Noise Schedule for Diffusion Trainingcited by 39
  13. 2024FreeEdit: Mask-free Reference-based Image Editing with Multi-modal Instructioncited by 36
  14. 2024Unveil Inversion and Invariance in Flow Transformer for Versatile Image Editingcited by 34
  15. 2024InstantSwap: Fast Customized Concept Swapping across Sharp Shape Differencescited by 34
  16. 2025Selftok: Discrete Visual Tokens of Autoregression, by Diffusion, and for Reasoningcited by 33
  17. 2025FireEdit: Fine-grained Instruction-based Image Editing via Region-aware Vision Language Modelcited by 32
  18. 2024PixWizard: Versatile Image-to-Image Visual Assistant with Open-Language Instructionscited by 31
  19. 2024BrushEdit: All-In-One Image Inpainting and Editingcited by 31
  20. 2023InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalistscited by 30
  21. 2024EmoEdit: Evoking Emotions through Image Manipulationcited by 30
  22. 2024MagicQuill: An Intelligent Interactive Image Editing Systemcited by 30
  23. 2024DreamOmni: Unified Image Generation and Editingcited by 30
  24. 2023UniGS: Unified Representation for Image Generation and Segmentationcited by 29
  25. 2024Conditional Image Synthesis with Diffusion Models: A Surveycited by 29
  26. 2023LIME: Localized Image Editing via Attention Regularization in Diffusion Modelscited by 27
  27. 2023SPIRE: Semantic Prompt-Driven Image Restorationcited by 24
  28. 2024Lumen: Unleashing Versatile Vision-Centric Capabilities of Large Multimodal Modelscited by 24
  29. 2025RORem: Training a Robust Object Remover with Human-in-the-Loopcited by 23
  30. 2025StyleSSP: Sampling StartPoint Enhancement for Training-free Diffusion-based Method for Style Transfercited by 23
  31. 2023Hulk: A Universal Knowledge Translator for Human-Centric Taskscited by 22
  32. 2024LaVin-DiT: Large Vision Diffusion Transformercited by 21
  33. 2024SemFlow: Binding Semantic Segmentation and Image Synthesis via Rectified Flowcited by 20
  34. 2024Multi-Reward as Condition for Instruction-based Image Editingcited by 20
  35. 2023VIDiff: Translating Videos via Multi-Modal Instructions with Diffusion Modelscited by 19
  36. 2025PlanGen: Towards Unified Layout Planning and Image Generation in Auto-Regressive Vision Language Modelscited by 18
  37. 2025SuperEdit: Rectifying and Facilitating Supervision for Instruction-Based Image Editingcited by 18
  38. 2024Visual-O1: Understanding Ambiguous Instructions via Multi-modal Multi-turn Chain-of-thoughts Reasoningcited by 17

50 most cited of 164 All 164 (JSON)