mlsanityData to 7 Oct 2026

InstructPix2Pix: Learning to Follow Image Editing Instructions

arXiv 2211.09800 ↗First posted 17 Nov 2022cs.CV cs.AI cs.CLcited by 2,591

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

We propose a method for editing images from human instructions: given an input image and a written instruction that tells the model what to do, our model follows these instructions to edit the image. To obtain training data for this problem, we combine the knowledge of two large pretrained models -- a language model (GPT-3) and a text-to-image model (Stable Diffusion) -- to generate a large dataset of image editing examples. Our conditional diffusion model, InstructPix2Pix, is trained on our generated data, and generalizes to real images and user-written instructions at inference time. Since it performs edits in the forward pass and does not require per example fine-tuning or inversion, our model edits images quickly, in a matter of seconds. We show compelling editing results for a diverse collection of input images and written instructions.

Key facts. Cited by 2,591 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 68 of its 75 references that are in the map. First posted on arXiv 17 Nov 2022.

Builds on

68 of 75 references.

Closest ancestors · 10

Same direction, newest first

  1. 2022Imagic: Text-Based Real Image Editing with Diffusion Models17 Oct
  2. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
  3. 2022Compositional Visual Generation with Composable Diffusion Models3 Jun
  4. 2022Pretraining is All You Need for Image-to-Image Translation25 May
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  7. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec
  8. 2021Blended Diffusion for Text-driven Editing of Natural Images29 Nov
  9. 2021Palette: Image-to-Image Diffusion Models10 Nov
  10. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation6 Oct

Other references · 58

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  8. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  9. 2022Classifier-Free Diffusion Guidancecited by 5,717
  10. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  12. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
38 more
  1. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  2. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  3. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  4. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  5. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  6. 2022Video Diffusion Modelscited by 2,280
  7. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  8. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  9. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  10. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
  11. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  12. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  13. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  14. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  15. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  16. 2018Multimodal Unsupervised Image-to-Image Translationcited by 795
  17. 2020Encoding in Style: a StyleGAN Encoder for Image-to-Image Translationcited by 727
  18. 2017Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networkscited by 707
  19. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  20. 2019Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?cited by 675
  21. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  22. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  23. 2021Designing an Encoder for StyleGAN Image Manipulationcited by 542
  24. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  25. 2015A Neural Algorithm of Artistic Stylecited by 480
  26. 2019Image2StyleGAN++: How to Edit the Embedded Images?cited by 356
  27. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  28. 2016Learning from Simulated and Unsupervised Images through Adversarial Trainingcited by 287
  29. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  30. 2019Few-Shot Unsupervised Image-to-Image Translationcited by 222
  31. 2021HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editingcited by 208
  32. 2021Few-shot Image Generation via Cross-domain Correspondencecited by 206
  33. 2021CLIPstyler: Image Style Transfer with a Single Text Conditioncited by 203
  34. 2021StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generatorscited by 164
  35. 2021ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmeticcited by 150
  36. 2019Classification Accuracy Score for Conditional Generative Modelscited by 134
  37. 2022BigDatasetGAN: Synthesizing ImageNet with Pixel-wise Annotationscited by 99
  38. 2021Repurposing GANs for One-shot Semantic Part Segmentationcited by 88

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 444; 2024: 745; 2025: 800; 2026: 598.

2,591 papers.

Sort:

By area:

  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  3. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  4. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,483
  5. 2025Qwen-Image Technical Reportcited by 973
  6. 2025FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Spacecited by 960
  7. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 867
  8. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
  9. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Modelscited by 696
  10. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 666
  11. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  12. 2024Improved Distribution Matching Distillation for Fast Image Synthesiscited by 648
38 more
  1. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  2. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  3. 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 615
  4. 2023Segment Everything Everywhere All at Oncecited by 546
  5. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generationcited by 529
  6. 2023Zero-shot Image-to-Image Translationcited by 503
  7. 2023Instruct-NeRF2NeRF: Editing 3D Scenes with Instructionscited by 482
  8. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 479
  9. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 468
  10. 2025VACE: All-in-One Video Creation and Editingcited by 467
  11. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 465
  12. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  13. 2023BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editingcited by 438
  14. 2023Emergent Correspondence from Image Diffusioncited by 437
  15. 2025Step1X-Edit: A Practical Framework for General Image Editingcited by 417
  16. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 396
  17. 2024OmniGen: Unified Image Generationcited by 388
  18. 2023Gaussian Grouping: Segment and Edit Anything in 3D Scenescited by 387
  19. 2023Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Modelscited by 358
  20. 2023InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuningcited by 343
  21. 2025Hunyuan3D 2.0: Scaling Diffusion Models for High Resolution Textured 3D Assets Generationcited by 342
  22. 2023SVDiff: Compact Parameter Space for Diffusion Fine-Tuningcited by 338
  23. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 337
  24. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 337
  25. 2023GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splattingcited by 331
  26. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 328
  27. 2023DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Datacited by 322
  28. 2023Emu Edit: Precise Image Editing via Recognition and Generation Taskscited by 321
  29. 2023Pix2Video: Video Editing using Image Diffusioncited by 318
  30. 2023ReconFusion: 3D Reconstruction with Diffusion Priorscited by 317
  31. 2023Video-P2P: Video Editing with Cross-attention Controlcited by 312
  32. 2023Diffusion Self-Guidance for Controllable Image Generationcited by 305
  33. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  34. 2023DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editingcited by 293
  35. 2023The Stable Signature: Rooting Watermarks in Latent Diffusion Modelscited by 289
  36. 2023Human Preference Score: Better Aligning Text-to-Image Models with Human Preferencecited by 284
  37. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Modelscited by 284
  38. 2024Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representationscited by 279

50 most cited of 2,591 All 2,591 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026How Well Do Models Follow Visual Instructions? VIBE: A Systematic Benchmark for Visual Instruction-Driven Image Editing
  2. 2025PIXELS: Progressive Image Xemplar-based Editing with Latent Surgery
  3. 2025X2Edit: Revisiting Arbitrary-Instruction Image Editing through Self-Constructed Data and Task-Aware Representation Learning
  4. 2022DiffEdit: Diffusion-based semantic image editing with mask guidance
  5. 2025Self-Attention Decomposition For Training Free Diffusion Editing
  6. 2026Scalable Minimal-Change Learning for Controllable Image Editing
  7. 2026MCIE: Multimodal LLM-Driven Complex Instruction Image Editing with Spatial Guidance