mlsanityData to 14 Oct 2026

Visual Instruction Inversion: Image Editing via Visual Prompting

arXiv 2307.14331 ↗First posted 26 Jul 2023cs.CVcited by 26

Not on the map. Why?

Abstract (arXiv)

Text-conditioned image editing has emerged as a powerful tool for editing images. However, in many situations, language can be ambiguous and ineffective in describing specific image edits. When faced with such challenges, visual prompts can be a more informative and intuitive way to convey ideas. We present a method for image editing via visual prompting. Given pairs of example that represent the "before" and "after" images of an edit, our goal is to learn a text-based editing direction that can be used to perform the same edit on new images. We leverage the rich, pretrained editing capabilities of text-to-image diffusion models by inverting visual prompts into editing instructions. Our results show that with just one example pair, we can achieve competitive results compared to state-of-the-art text-conditioned image editing frameworks.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Builds on 50 of its 54 references that are in the map. First posted on arXiv 26 Jul 2023.

Builds on

50 of 54 references.

References · 50

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  8. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  9. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  10. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  11. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  12. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
38 more
  1. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  2. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  3. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  4. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  5. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  6. 2021Palette: Image-to-Image Diffusion Modelscited by 1,300
  7. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  8. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,179
  9. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,178
  10. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 991
  11. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 931
  12. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 781
  13. 2020Encoding in Style: a StyleGAN Encoder for Image-to-Image Translationcited by 727
  14. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  15. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulationcited by 642
  16. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 577
  17. 2021ClipCap: CLIP Prefix for Image Captioningcited by 559
  18. 2021Designing an Encoder for StyleGAN Image Manipulationcited by 544
  19. 2023Zero-shot Image-to-Image Translationcited by 505
  20. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  21. 2021Pivotal Tuning for Latent-based Editing of Real Imagescited by 427
  22. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 336
  23. 2015Skip-Thought Vectorscited by 322
  24. 2022Images Speak in Images: A Generalist Painter for In-Context Visual Learningcited by 318
  25. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  26. 2022Visual Prompting via Image Inpaintingcited by 250
  27. 2022Optimizing Prompts for Text-to-Image Generationcited by 207
  28. 2023SegGPT: Segmenting Everything In Contextcited by 194
  29. 2022ReCo: Region-Controlled Text-to-Image Generationcited by 179
  30. 2022SINE: SINgle Image Editing with Text-to-Image Diffusion Modelscited by 159
  31. 2023What Makes Good Examples for Visual In-Context Learning?cited by 150
  32. 2017Visual Attribute Transfer through Deep Image Analogycited by 123
  33. 2023In-Context Learning Unlocked for Diffusion Modelscited by 104
  34. 2021HyperInverter: Improving StyleGAN Inversion via Hypernetworkcited by 85
  35. 2023ReVersion: Diffusion-Based Relation Inversion from Imagescited by 82
  36. 2022Investigating Prompt Engineering in Diffusion Modelscited by 58
  37. 2022Third Time's the Charm? Image and Video Editing with StyleGAN3cited by 54
  38. 2022Self-Conditioned Generative Adversarial Networks for Image Editingcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 13; 2025: 4; 2026: 4.

26 papers.

Sort:

By area:

  1. 2024Large Motion Model for Unified Multi-Modal Motion Generationcited by 65
  2. 2024Visual Style Prompting with Swapping Self-Attentioncited by 63
  3. 2024Yo'LLaVA: Your Personalized Language and Vision Assistantcited by 62
  4. 2023Generative Models: What Do They Know? Do They Know Things? Let's Find Out!cited by 50
  5. 2023MEVG: Multi-event Video Generation with Text-to-Video Modelscited by 40
  6. 2023Exploiting Diffusion Prior for Generalizable Dense Predictioncited by 38
  7. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  8. 2024A Survey on Data Augmentation in Large Model Eracited by 31
  9. 2024Can MLLMs Perform Text-to-Image In-Context Learning?cited by 13
  10. 2024Unleashing In-context Learning of Autoregressive Models for Few-shot Image Manipulationcited by 13
  11. 2023MoEController: Instruction-based Arbitrary Image Manipulation with Mixture-of-Expert Controllerscited by 12
  12. 2024InstructGIE: Towards Generalizable Image Editingcited by 12
14 more
  1. 2024AdvI2I: Adversarial Image Attack on Image-to-Image Diffusion modelscited by 10
  2. 2025PairEdit: Learning Semantic Variations for Exemplar-based Image Editingcited by 8
  3. 2024LoRA of Change: Learning to Generate LoRA for the Editing Instruction from A Single Before-After Image Paircited by 6
  4. 2025StyleKeeper: Prevent Content Leakage using Negative Visual Query Guidancecited by 6
  5. 2024Edit One for All: Interactive Batch Image Editingcited by 5
  6. 2025GuardDoor: Safeguarding Against Malicious Diffusion Editing via Protective Backdoorscited by 5
  7. 2026Instruction-based Image Editing with Planning, Reasoning, and Generationcited by 4
  8. 2024ED-SAM: An Efficient Diffusion Sampling Approach to Domain Generalization in Vision-Language Foundation Modelscited by 2
  9. 2025Personalized Image Filter: Mastering Your Photographic Stylecited by 2
  10. 2026Delta-Adapter: Scalable Exemplar-Based Image Editing with Single-Pair Supervisioncited by 1
  11. 2023Textual and Visual Prompt Fusion for Image Editing via Step-Wise Alignmentcited by 0
  12. 2024OSTAF: A One-Shot Tuning Method for Improved Attribute-Focused T2I Personalizationcited by 0
  13. 2026TransPhy: Visual In-Context Learning for Physically Grounded Image Editingcited by 0
  14. 2026GUIDE: Guiding Internal Evidence with Language Instructionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Language-Free Generative Editing from One Visual Example
  2. 2026EditTransfer++: Toward Faithful and Efficient Visual-Prompt-Guided Image Editing
  3. 2023StyleDiffusion: Prompt-Embedding Inversion for Text-Based Editing
  4. 2023Prompt Tuning Inversion for Text-Driven Image Editing Using Diffusion Models
  5. 2024Learn Once, Edit Anywhere: Visual Direction Transfer for Diffusion Models
  6. 2024Prompt Augmentation for Self-supervised Text-guided Image Manipulation
  7. 2023User-friendly Image Editing with Minimal Text Input: Leveraging Captioning and Injection Techniques
  8. 2025RelationAdapter: Learning and Transferring Visual Relation with Diffusion Transformers