mlsanityData to 14 Oct 2026

ZONE: Zero-Shot Instruction-Guided Local Editing

arXiv 2312.16794 ↗First posted 28 Dec 2023cs.CVcited by 62

On the map: Multimodal LLMs / vision-language models → Image editing and quality assessment

also in 1 more
Abstract (arXiv)

Recent advances in vision-language models like Stable Diffusion have shown remarkable power in creative image synthesis and editing.However, most existing text-to-image editing methods encounter two obstacles: First, the text prompt needs to be carefully crafted to achieve good results, which is not intuitive or user-friendly. Second, they are insensitive to local edits and can irreversibly affect non-edited regions, leaving obvious editing traces. To tackle these problems, we propose a Zero-shot instructiON-guided local image Editing approach, termed ZONE. We first convert the editing intent from the user-provided instruction (e.g., "make his tie blue") into specific image editing regions through InstructPix2Pix. We then propose a Region-IoU scheme for precise image layer extraction from an off-the-shelf segment model. We further develop an edge smoother based on FFT for seamless blending between the layer and the image.Our method allows for arbitrary manipulation of a specific region with a single instruction while preserving the rest. Extensive experiments demonstrate that our ZONE achieves remarkable local editing results and user-friendliness, outperforming state-of-the-art methods. Code is available at https://github.com/lsl001006/ZONE.

Key facts. Cited by 62 papers in the map (data through 14 Oct 2026). Placed in Image editing and quality assessment, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 55 of its 63 references that are in the map. First posted on arXiv 28 Dec 2023.

Builds on

55 of 63 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Blended Latent Diffusion6 Jun

Other references · 54

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  7. 2020Denoising Diffusion Implicit Modelscited by 9,781
  8. 2023Segment Anythingcited by 8,234
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  12. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
38 more
  1. 2022Classifier-Free Diffusion Guidancecited by 5,847
  2. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  4. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  5. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  6. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  7. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  8. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  9. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  10. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  11. 2016Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Networkcited by 1,908
  12. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,430
  13. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
  14. 2021Palette: Image-to-Image Diffusion Modelscited by 1,300
  15. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  16. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,178
  17. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,163
  18. 2017StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translationcited by 1,039
  19. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 927
  20. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 906
  21. 2022DiffEdit: Diffusion-based semantic image editing with mask guidancecited by 647
  22. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulationcited by 642
  23. 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 624
  24. 2023Zero-shot Image-to-Image Translationcited by 505
  25. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  26. 2023ResShift: Efficient Diffusion Model for Image Super-resolution by Residual Shiftingcited by 365
  27. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  28. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  29. 2017Learning to Discover Cross-Domain Relations with Generative Adversarial Networkscited by 278
  30. 2023Implicit Diffusion Models for Continuous Super-Resolutioncited by 218
  31. 2021CLIPstyler: Image Style Transfer with a Single Text Conditioncited by 203
  32. 2023StyleDrop: Text-to-Image Generation in Any Stylecited by 199
  33. 2023HIVE: Harnessing Human Feedback for Instructional Visual Editingcited by 173
  34. 2022Pretraining is All You Need for Image-to-Image Translationcited by 160
  35. 2018Tell, Draw, and Repeat: Generating and Modifying Images Based on Continual Linguistic Instructioncited by 61
  36. 2022Self-Distilled StyleGAN: Towards Generation from Internet Photoscited by 37
  37. 2023Controllable Mind Visual Diffusion Modelcited by 30
  38. 2020Text as Neural Operator: Image Manipulation by Text Instructioncited by 26

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 19; 2025: 27; 2026: 16.

62 papers.

Sort:

By area:

  1. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  2. 2024Taming Rectified Flow for Inversion and Editingcited by 204
  3. 2025EasyControl: Adding Efficient and Flexible Control for Diffusion Transformercited by 119
  4. 2024Stable Flow: Vital Layers for Training-Free Image Editingcited by 113
  5. 2024Attentive Eraser: Unleashing Diffusion Model's Object Removal Potential via Self-Attention Redirection Guidancecited by 63
  6. 2025Any2AnyTryon: Leveraging Adaptive Position Embeddings for Versatile Virtual Clothing Taskscited by 63
  7. 2025UniCTokens: Boosting Personalized Understanding and Generation via Unified Concept Tokenscited by 56
  8. 2024Continuous, Subject-Specific Attribute Control in T2I Models by Identifying Semantic Directionscited by 41
  9. 2024EditWorld: Simulating World Dynamics for Instruction-Following Image Editingcited by 38
  10. 2024Stable-Hair: Real-World Hair Transfer via Diffusion Modelcited by 35
  11. 2025FireEdit: Fine-grained Instruction-based Image Editing via Region-aware Vision Language Modelcited by 32
  12. 2024MM-Soc: Benchmarking Multimodal Large Language Models in Social Media Platformscited by 26
38 more
  1. 2025PhotoDoodle: Learning Artistic Image Editing from Few-Shot Pairwise Datacited by 24
  2. 2025DNAEdit: Direct Noise Alignment for Text-Guided Rectified Flow Editingcited by 24
  3. 2025FreeFlux: Understanding and Exploiting Layer-Specific Roles in RoPE-Based MMDiT for Versatile Image Editingcited by 22
  4. 2024Trans4D: Realistic Geometry-Aware Transition for Compositional Text-to-4D Synthesiscited by 19
  5. 2025VINCIE: Unlocking In-context Image Editing from Videocited by 17
  6. 2024Exploring Low-Dimensional Subspaces in Diffusion Models for Controllable Image Editingcited by 15
  7. 2025SliderEdit: Continuous Image Editing with Fine-Grained Instruction Controlcited by 14
  8. 2025Bayesian Optimization for Controlled Image Editing via LLMscited by 11
  9. 2024InstructBrush: Learning Attention-based Instruction Optimization for Image Editingcited by 10
  10. 2024DiffuX2CT: Diffusion Learning to Reconstruct CT Images from Biplanar X-Rayscited by 9
  11. 2025Instruct-CLIP: Improving Instruction-Guided Image Editing with Automated Data Refinement Using Contrastive Learningcited by 9
  12. 2024Model Inversion Attacks Through Target-Specific Conditional Diffusion Modelscited by 8
  13. 2024Dual-Schedule Inversion: Training- and Tuning-Free Inversion for Real Image Editingcited by 8
  14. 2025Beyond Editing Pairs: Fine-Grained Instructional Image Editing via Multi-Scale Learnable Regionscited by 7
  15. 2025EditInfinity: Image Editing with Binary-Quantized Generative Modelscited by 7
  16. 2025Early Timestep Zero-Shot Candidate Selection for Instruction-Guided Image Editingcited by 5
  17. 2025ManipShield: A Unified Framework for Image Manipulation Detection, Localization and Explanationcited by 5
  18. 2024InstructHumans: Editing Animated 3D Human Textures with Instructionscited by 4
  19. 2026EditHF-1M: A Million-Scale Rich Human Preference Feedback for Image Editingcited by 4
  20. 2024NeIn: Telling What You Don't Wantcited by 3
  21. 2025OmniText: A Training-Free Generalist for Controllable Text-Image Manipulationcited by 3
  22. 2026Research on World Models Is Not Merely Injecting World Knowledge into Specific Taskscited by 3
  23. 2026Evaluating Image Editing with LLMs: A Comprehensive Benchmark and Intermediate-Layer Probing Approachcited by 3
  24. 2024Composing Parts for Expressive Object Generationcited by 2
  25. 2024Varying Manifolds in Diffusion: From Time-varying Geometries to Visual Saliencycited by 2
  26. 2025CPAM: Context-Preserving Adaptive Manipulation for Zero-Shot Real Image Editingcited by 2
  27. 2025Stable-Hair v2: Real-World Hair Transfer via Multiple-View Diffusion Modelcited by 2
  28. 2025CannyEdit: Selective Canny Control and Dual-Prompt Guidance for Training-Free Image Editingcited by 2
  29. 2025CogniEdit: Dense Gradient Flow Optimization for Fine-Grained Image Editingcited by 2
  30. 2024HyperGAN-CLIP: A Unified Framework for Domain Adaptation, Image Synthesis and Manipulationcited by 1
  31. 2025InstructRestore: Region-Customized Image Restoration with Human Instructionscited by 1
  32. 2025CAMILA: Context-Aware Masking for Image Editing with Language Alignmentcited by 1
  33. 2025NeXT-IMDL: Build Benchmark for NeXT-Generation Image Manipulation Detection & Localizationcited by 1
  34. 2026Fine-Grained Human Pose Editing Assessment via Layer-Selective MLLMscited by 1
  35. 2026Semantic Granularity Navigation in Image Editingcited by 1
  36. 2026Beyond Starry Night: Shortcut-Aware Control-State Planning for Artist-Grounded Text to Image Generationcited by 1
  37. 2025MILD: Multi-Layer Diffusion Strategy for Complex and Precise Multi-IP Aware Human Erasingcited by 0
  38. 2025Q-Align: Alleviating Attention Leakage in Zero-Shot Appearance Transfer via Query-Query Alignmentcited by 0

50 most cited of 62 All 62 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026WhereEdit: Mask-aware Local Latent Editing for One-Step Image Editing
  2. 2026MRT: Masked Region Transformer for Layered Image Generation and Editing at Scale
  3. 2024LoMOE: Localized Multi-Object Editing via Multi-Diffusion
  4. 2026SI-Edit: Toward Sketch-Instruction Guided Local Image Editing with Pixel-Level Precision
  5. 2023Region-Aware Diffusion for Zero-shot Text-driven Image Editing
  6. 2023LayerDiffusion: Layered Controlled Image Editing with Diffusion Models
  7. 2026Rethinking Where to Edit: Task-Aware Localization for Instruction-Based Image Editing
  8. 2023MAG-Edit: Localized Image Editing in Complex Scenarios via Mask-Based Attention-Adjusted Guidance