# Visual Instruction Inversion: Image Editing via Visual Prompting

arXiv: https://arxiv.org/abs/2307.14331 · HTML: https://mlsanity.com/paper/2307.14331/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 54 references that are in the map.
- First posted on arXiv 26 Jul 2023.

## Builds on (most cited first, 30 of 50)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022)
- [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers](https://mlsanity.com/paper/2211.01324/) (2022)
- [GLIGEN: Open-Set Grounded Text-to-Image Generation](https://mlsanity.com/paper/2301.07093/) (2023)
- [Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation](https://mlsanity.com/paper/2008.00951/) (2020)
- [AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks](https://mlsanity.com/paper/1711.10485/) (2017)
- [DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation](https://mlsanity.com/paper/2110.02711/) (2021)
- [StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks](https://mlsanity.com/paper/1612.03242/) (2016)
- [ClipCap: CLIP Prefix for Image Captioning](https://mlsanity.com/paper/2111.09734/) (2021)
- [Designing an Encoder for StyleGAN Image Manipulation](https://mlsanity.com/paper/2102.02766/) (2021)

## Built on by (most cited first, 26 of 26)

- [Large Motion Model for Unified Multi-Modal Motion Generation](https://mlsanity.com/paper/2404.01284/) (2024)
- [Visual Style Prompting with Swapping Self-Attention](https://mlsanity.com/paper/2402.12974/) (2024)
- [Yo'LLaVA: Your Personalized Language and Vision Assistant](https://mlsanity.com/paper/2406.09400/) (2024)
- [Generative Models: What Do They Know? Do They Know Things? Let's Find Out!](https://mlsanity.com/paper/2311.17137/) (2023)
- [MEVG: Multi-event Video Generation with Text-to-Video Models](https://mlsanity.com/paper/2312.04086/) (2023)
- [Exploiting Diffusion Prior for Generalizable Dense Prediction](https://mlsanity.com/paper/2311.18832/) (2023)
- [LLMs Meet Multimodal Generation and Editing: A Survey](https://mlsanity.com/paper/2405.19334/) (2024)
- [A Survey on Data Augmentation in Large Model Era](https://mlsanity.com/paper/2401.15422/) (2024)
- [Can MLLMs Perform Text-to-Image In-Context Learning?](https://mlsanity.com/paper/2402.01293/) (2024)
- [Unleashing In-context Learning of Autoregressive Models for Few-shot Image Manipulation](https://mlsanity.com/paper/2412.01027/) (2024)
- [MoEController: Instruction-based Arbitrary Image Manipulation with Mixture-of-Expert Controllers](https://mlsanity.com/paper/2309.04372/) (2023)
- [InstructGIE: Towards Generalizable Image Editing](https://mlsanity.com/paper/2403.05018/) (2024)
- [AdvI2I: Adversarial Image Attack on Image-to-Image Diffusion models](https://mlsanity.com/paper/2410.21471/) (2024)
- [PairEdit: Learning Semantic Variations for Exemplar-based Image Editing](https://mlsanity.com/paper/2506.07992/) (2025)
- [LoRA of Change: Learning to Generate LoRA for the Editing Instruction from A Single Before-After Image Pair](https://mlsanity.com/paper/2411.19156/) (2024)
- [StyleKeeper: Prevent Content Leakage using Negative Visual Query Guidance](https://mlsanity.com/paper/2510.06827/) (2025)
- [Edit One for All: Interactive Batch Image Editing](https://mlsanity.com/paper/2401.10219/) (2024)
- [GuardDoor: Safeguarding Against Malicious Diffusion Editing via Protective Backdoors](https://mlsanity.com/paper/2503.03944/) (2025)
- [Instruction-based Image Editing with Planning, Reasoning, and Generation](https://mlsanity.com/paper/2602.22624/) (2026)
- [ED-SAM: An Efficient Diffusion Sampling Approach to Domain Generalization in Vision-Language Foundation Models](https://mlsanity.com/paper/2406.01432/) (2024)
- [Personalized Image Filter: Mastering Your Photographic Style](https://mlsanity.com/paper/2510.16791/) (2025)
- [Delta-Adapter: Scalable Exemplar-Based Image Editing with Single-Pair Supervision](https://mlsanity.com/paper/2605.07940/) (2026)
- [Textual and Visual Prompt Fusion for Image Editing via Step-Wise Alignment](https://mlsanity.com/paper/2308.15854/) (2023)
- [OSTAF: A One-Shot Tuning Method for Improved Attribute-Focused T2I Personalization](https://mlsanity.com/paper/2403.11053/) (2024)
- [TransPhy: Visual In-Context Learning for Physically Grounded Image Editing](https://mlsanity.com/paper/2608.24119/) (2026)
- [GUIDE: Guiding Internal Evidence with Language Instructions](https://mlsanity.com/paper/2608.30712/) (2026)

## Abstract (arXiv)

Text-conditioned image editing has emerged as a powerful tool for editing images. However, in many situations, language can be ambiguous and ineffective in describing specific image edits. When faced with such challenges, visual prompts can be a more informative and intuitive way to convey ideas. We present a method for image editing via visual prompting. Given pairs of example that represent the "before" and "after" images of an edit, our goal is to learn a text-based editing direction that can be used to perform the same edit on new images. We leverage the rich, pretrained editing capabilities of text-to-image diffusion models by inverting visual prompts into editing instructions. Our results show that with just one example pair, we can achieve competitive results compared to state-of-the-art text-conditioned image editing frameworks.