mlsanityData to 7 Oct 2026

Guiding Instruction-based Image Editing via Multimodal Large Language Models

arXiv 2309.17102 ↗First posted 29 Sep 2023cs.CVcited by 182

On the map: Multimodal LLMs / vision-language models → Image editing and quality assessment

Abstract (arXiv)

Instruction-based image editing improves the controllability and flexibility of image manipulation via natural commands without elaborate descriptions or regional masks. However, human instructions are sometimes too brief for current methods to capture and follow. Multimodal large language models (MLLMs) show promising capabilities in cross-modal understanding and visual-aware response generation via LMs. We investigate how MLLMs facilitate edit instructions and present MLLM-Guided Image Editing (MGIE). MGIE learns to derive expressive instructions and provides explicit guidance. The editing model jointly captures this visual imagination and performs manipulation through end-to-end training. We evaluate various aspects of Photoshop-style modification, global photo optimization, and local editing. Extensive experimental results demonstrate that expressive instructions are crucial to instruction-based image editing, and our MGIE can lead to a notable improvement in automatic metrics and human evaluation while maintaining competitive inference efficiency.

Key facts. Cited by 182 papers in the map (data through 7 Oct 2026). Placed in Image editing and quality assessment, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 60 of its 67 references that are in the map. First posted on arXiv 29 Sep 2023.

Builds on

60 of 67 references.

References · 60

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  12. 2022Classifier-Free Diffusion Guidancecited by 5,717
38 more
  1. 2022Generative Adversarial Networkscited by 5,603
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  7. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  8. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  9. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  10. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  11. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  12. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,168
  13. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  14. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  15. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 924
  16. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  17. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  18. 2016Generative Adversarial Text to Image Synthesiscited by 735
  19. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  20. 2022DiffEdit: Diffusion-based semantic image editing with mask guidancecited by 643
  21. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulationcited by 640
  22. 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 615
  23. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  24. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  25. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  26. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  27. 2023BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editingcited by 438
  28. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  29. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 337
  30. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  31. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  32. 2023Generating Images with Multimodal Language Modelscited by 259
  33. 2023LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Modelscited by 242
  34. 2022Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpaintingcited by 229
  35. 2023HIVE: Harnessing Human Feedback for Instructional Visual Editingcited by 172
  36. 2019ManiGAN: Text-Guided Image Manipulationcited by 156
  37. 2023Emu: Generative Pretraining in Multimodalitycited by 156
  38. 2018Text-Adaptive Generative Adversarial Networks: Manipulating Images with Natural Languagecited by 90

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 56; 2025: 68; 2026: 51.

182 papers.

Sort:

By area:

  1. 2024AnyEdit: Mastering Unified High-Quality Image Editing for Any Ideacited by 235
  2. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  3. 2024Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMscited by 224
  4. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  5. 2025In-Context Edit: Enabling Instructional Image Editing with In-Context Generation in Large Scale Diffusion Transformercited by 199
  6. 2023SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Modelscited by 174
  7. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
  8. 2024GenArtist: Multimodal LLM as an Agent for Unified Image Generation and Editingcited by 132
  9. 2025GoT: Unleashing Reasoning Capability of Multimodal Large Language Model for Visual Generation and Editingcited by 103
  10. 2024UniReal: Universal Image Generation and Editing via Learning Real-world Dynamicscited by 100
  11. 2024SEED-Data-Edit Technical Report: A Hybrid Dataset for Instructional Image Editingcited by 97
  12. 2025VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Modelscited by 88
38 more
  1. 2024ObjectDrop: Bootstrapping Counterfactuals for Photorealistic Object Removal and Insertioncited by 71
  2. 2024MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMscited by 62
  3. 2023Focus on Your Instruction: Fine-grained and Multi-instruction Image Editing by Attention Modulationcited by 61
  4. 2024I2EBench: A Comprehensive Benchmark for Instruction-based Image Editingcited by 61
  5. 2025Se~norita-2M: A High-Quality Instruction-based Dataset for General Video Editing by Video Specialistscited by 61
  6. 2025Pico-Banana-400K: A Large-Scale Dataset for Text-Guided Image Editingcited by 58
  7. 2023Relightful Harmonization: Lighting-aware Portrait Background Replacementcited by 56
  8. 2024MineDreamer: Learning to Follow Instructions via Chain-of-Imagination for Simulated-World Controlcited by 52
  9. 2024How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on Deceptive Promptscited by 47
  10. 2024Towards Semantic Equivalence of Tokenization in Multimodal LLMcited by 45
  11. 2024PromptFix: You Prompt and We Fix the Photocited by 40
  12. 2025EditAR: Unified Conditional Generation with Autoregressive Modelscited by 40
  13. 2024Efficient Diffusion Models: A Comprehensive Survey from Principles to Practicescited by 39
  14. 2024HiCo: Hierarchical Controllable Diffusion Model for Layout-to-image Generationcited by 39
  15. 2025VEGGIE: Instructional Editing and Reasoning Video Concepts with Grounded Generationcited by 39
  16. 2025JarvisArt: Liberating Human Artistic Creativity via an Intelligent Photo Retouching Agentcited by 39
  17. 2025OmniPaint: Mastering Object-Oriented Editing via Disentangled Insertion-Removal Inpaintingcited by 38
  18. 2024EditWorld: Simulating World Dynamics for Instruction-Following Image Editingcited by 37
  19. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  20. 2024FreeEdit: Mask-free Reference-based Image Editing with Multi-modal Instructioncited by 35
  21. 2025OmniEraser: Remove Objects and Their Effects in Images with Paired Video-Frame Datacited by 35
  22. 2024Generative Image Layer Decomposition with Visual Effectscited by 33
  23. 2026ImageWAM: Do World Action Models Really Need Video Generation, or Just Image Editing?cited by 33
  24. 2024Learning Action and Reasoning-Centric Image Editing from Videos and Simulationscited by 32
  25. 2024BrushEdit: All-In-One Image Inpainting and Editingcited by 31
  26. 2025FireEdit: Fine-grained Instruction-based Image Editing via Region-aware Vision Language Modelcited by 31
  27. 2025Selftok: Discrete Visual Tokens of Autoregression, by Diffusion, and for Reasoningcited by 31
  28. 2024Auto-Encoding Morph-Tokens for Multimodal LLMcited by 30
  29. 2024MagicQuill: An Intelligent Interactive Image Editing Systemcited by 30
  30. 2024Conditional Image Synthesis with Diffusion Models: A Surveycited by 29
  31. 2024DreamOmni: Unified Image Generation and Editingcited by 29
  32. 2023EditShield: Protecting Unauthorized Image Editing by Instruction-guided Diffusion Modelscited by 27
  33. 2023LIME: Localized Image Editing via Attention Regularization in Diffusion Modelscited by 27
  34. 2025Generative Multimodal Pretraining with Discrete Diffusion Timestep Tokenscited by 27
  35. 2024Octopus: On-device language model for function calling of software APIscited by 26
  36. 2025textttComplex-Edit: CoT-Like Instruction Generation for Complexity-Controllable Image Editing Benchmarkcited by 25
  37. 2023InstructAny2Pix: Flexible Visual Editing via Multimodal Instruction Followingcited by 24
  38. 2025Visual Autoregressive Modeling for Instruction-Guided Image Editingcited by 21

50 most cited of 182 All 182 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026MCIE: Multimodal LLM-Driven Complex Instruction Image Editing with Spatial Guidance
  2. 2025Lego-Edit: A General Image Editing Framework with Model-Level Bricks and MLLM Builder
  3. 2025MIND-Edit: MLLM Insight-Driven Editing via Language-Vision Projection
  4. 2024Instruction-Guided Editing Controls for Images and Multimedia: A Survey in LLM era
  5. 2025InstructX: Towards Unified Visual Editing with MLLM Guidance
  6. 2025ReasonEdit: Towards Reasoning-Enhanced Image Editing Models
  7. 2026MSRAMIE: Multimodal Structured Reasoning Agent for Multi-instruction Image Editing
  8. 2026How Well Do Models Follow Visual Instructions? VIBE: A Systematic Benchmark for Visual Instruction-Driven Image Editing