mlsanityData to 7 Oct 2026

SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Models

arXiv 2312.06739 ↗First posted 11 Dec 2023cs.CVcited by 174

On the map: Multimodal LLMs / vision-language models → Image editing and quality assessment

also in 1 more
Abstract (arXiv)

Current instruction-based editing methods, such as InstructPix2Pix, often fail to produce satisfactory results in complex scenarios due to their dependence on the simple CLIP text encoder in diffusion models. To rectify this, this paper introduces SmartEdit, a novel approach to instruction-based image editing that leverages Multimodal Large Language Models (MLLMs) to enhance their understanding and reasoning capabilities. However, direct integration of these elements still faces challenges in situations requiring complex reasoning. To mitigate this, we propose a Bidirectional Interaction Module that enables comprehensive bidirectional information interactions between the input image and the MLLM output. During training, we initially incorporate perception data to boost the perception and understanding capabilities of diffusion models. Subsequently, we demonstrate that a small amount of complex instruction editing data can effectively stimulate SmartEdit's editing capabilities for more complex instructions. We further construct a new evaluation dataset, Reason-Edit, specifically tailored for complex instruction-based image editing. Both quantitative and qualitative results on this evaluation dataset indicate that our SmartEdit surpasses previous methods, paving the way for the practical application of complex instruction-based image editing.

Key facts. Cited by 174 papers in the map (data through 7 Oct 2026). Placed in Image editing and quality assessment, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 47 references that are in the map. First posted on arXiv 11 Dec 2023.

Builds on

38 of 47 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Guiding Instruction-based Image Editing via Multimodal Large Language Models29 Sep

Other references · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  9. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  10. 2023Segment Anythingcited by 8,158
  11. 2023Visual Instruction Tuningcited by 7,792
  12. 2020End-to-End Object Detection with Transformerscited by 7,384
25 more
  1. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  2. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2022Classifier-Free Diffusion Guidancecited by 5,717
  5. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  7. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  8. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  9. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  10. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  11. 2016Modeling Context in Referring Expressionscited by 1,054
  12. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 897
  13. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  14. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  15. 2023MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editingcited by 722
  16. 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 615
  17. 2023Generating Images with Multimodal Language Modelscited by 259
  18. 2023GRES: Generalized Referring Expression Segmentationcited by 252
  19. 2022MAT: Mask-Aware Transformer for Large Hole Image Inpaintingcited by 216
  20. 2023Making LLaMA SEE and Draw with SEED Tokenizercited by 184
  21. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuningcited by 168
  22. 2023InstructDiffusion: A Generalist Modeling Interface for Vision Taskscited by 162
  23. 2023Direct Inversion: Boosting Diffusion-based Editing with 3 Lines of Codecited by 145
  24. 2023Planting a SEED of Vision in Large Language Modelcited by 140
  25. 2023Semi-supervised Parametric Real-world Image Harmonizationcited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 38; 2025: 78; 2026: 57.

174 papers.

Sort:

By area:

  1. 2025Step1X-Edit: A Practical Framework for General Image Editingcited by 417
  2. 2025ImgEdit: A Unified Image Editing Dataset and Benchmarkcited by 225
  3. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  4. 2025In-Context Edit: Enabling Instructional Image Editing with In-Context Generation in Large Scale Diffusion Transformercited by 199
  5. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
  6. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  7. 2024GenArtist: Multimodal LLM as an Agent for Unified Image Generation and Editingcited by 132
  8. 2025HiDream-I1: A High-Efficient Image Generative Foundation Model with Sparse Diffusion Transformercited by 131
  9. 2025GoT: Unleashing Reasoning Capability of Multimodal Large Language Model for Visual Generation and Editingcited by 103
  10. 2025LongCat-Image Technical Reportcited by 92
  11. 2025VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Modelscited by 88
  12. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
38 more
  1. 2025KRIS-Bench: Benchmarking Next-Level Intelligent Image Editing Modelscited by 69
  2. 2023MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perceptioncited by 67
  3. 2024IMPRINT: Generative Object Compositing by Learning Identity-Preserving Representationcited by 64
  4. 2025InsViE-1M: Effective Instruction-based Video Editing with Elaborate Dataset Constructioncited by 62
  5. 2024I2EBench: A Comprehensive Benchmark for Instruction-based Image Editingcited by 61
  6. 2025Pico-Banana-400K: A Large-Scale Dataset for Text-Guided Image Editingcited by 58
  7. 2025GPT-ImgEval: A Comprehensive Benchmark for Diagnosing GPT4o in Image Generationcited by 57
  8. 2024ACE: All-round Creator and Editor Following Instructions via Diffusion Transformercited by 56
  9. 2024MineDreamer: Learning to Follow Instructions via Chain-of-Imagination for Simulated-World Controlcited by 52
  10. 2025SeedEdit 3.0: Fast and High-Quality Generative Image Editingcited by 52
  11. 2024Towards Semantic Equivalence of Tokenization in Multimodal LLMcited by 45
  12. 2025EditAR: Unified Conditional Generation with Autoregressive Modelscited by 40
  13. 2024EditWorld: Simulating World Dynamics for Instruction-Following Image Editingcited by 37
  14. 2024FreeEdit: Mask-free Reference-based Image Editing with Multi-modal Instructioncited by 35
  15. 2024Unveil Inversion and Invariance in Flow Transformer for Versatile Image Editingcited by 34
  16. 2024InstantSwap: Fast Customized Concept Swapping across Sharp Shape Differencescited by 34
  17. 2024BrushEdit: All-In-One Image Inpainting and Editingcited by 31
  18. 2025FireEdit: Fine-grained Instruction-based Image Editing via Region-aware Vision Language Modelcited by 31
  19. 2024MagicQuill: An Intelligent Interactive Image Editing Systemcited by 30
  20. 2025Does FLUX Already Know How to Perform Physically Plausible Image Composition?cited by 30
  21. 2024Conditional Image Synthesis with Diffusion Models: A Surveycited by 29
  22. 2024DreamOmni: Unified Image Generation and Editingcited by 29
  23. 2025IMAGHarmony: Controllable Image Editing with Consistent Object Quantity and Layoutcited by 28
  24. 2025textttComplex-Edit: CoT-Like Instruction Generation for Complexity-Controllable Image Editing Benchmarkcited by 25
  25. 2024F-HOI: Toward Fine-grained Semantic-Aligned 3D Human-Object Interactionscited by 24
  26. 2025UniREditBench: A Unified Reasoning-based Image Editing Benchmarkcited by 23
  27. 2024Multi-Reward as Condition for Instruction-based Image Editingcited by 20
  28. 2025VeOmni: Scaling Any Modality Model Training with Model-Centric Distributed Recipe Zoocited by 20
  29. 2024Edit Away and My Face Will not Stay: Personal Biometric Defense against Malicious Generative Editingcited by 19
  30. 2025SuperEdit: Rectifying and Facilitating Supervision for Instruction-Based Image Editingcited by 18
  31. 2024InsightEdit: Towards Better Instruction Following for Image Editingcited by 17
  32. 2024DIVE: Taming DINO for Subject-Driven Video Editingcited by 17
  33. 2025R3D2: Realistic 3D Asset Insertion via Diffusion for Autonomous Driving Simulationcited by 17
  34. 2024DiffSensei: Bridging Multi-Modal LLMs and Diffusion Models for Customized Manga Generationcited by 16
  35. 2025Factuality Matters: When Image Generation and Editing Meet Structured Visualscited by 16
  36. 2025ComfyMind: Toward General-Purpose Generation via Tree-Based Planning and Reactive Feedbackcited by 15
  37. 2025DreamPoster: A Unified Framework for Image-Conditioned Generative Poster Designcited by 15
  38. 2025Query-Kontext: An Unified Multimodal Model for Image Generation and Editingcited by 15

50 most cited of 174 All 174 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025ReasonEdit: Towards Reasoning-Enhanced Image Editing Models
  2. 2026MCIE: Multimodal LLM-Driven Complex Instruction Image Editing with Spatial Guidance
  3. 2026MSRAMIE: Multimodal Structured Reasoning Agent for Multi-instruction Image Editing
  4. 2024AnyEdit: Mastering Unified High-Quality Image Editing for Any Idea
  5. 2025InstructX: Towards Unified Visual Editing with MLLM Guidance
  6. 2025SmartFreeEdit: Mask-Free Spatial-Aware Image Editing with Complex Instruction Understanding
  7. 2024TIE: Revolutionizing Text-based Image Editing for Complex-Prompt Following and High-Fidelity Editing
  8. 2026TalkPhoto: A Versatile Training-Free Conversational Assistant for Intelligent Image Editing