# End-to-End Visual Editing with a Generatively Pre-Trained Artist

arXiv: https://arxiv.org/abs/2205.01668 · HTML: https://mlsanity.com/paper/2205.01668/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 74 of its 93 references that are in the map.
- First posted on arXiv 3 May 2022.

## Builds on (most cited first, 30 of 74)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Training Generative Adversarial Networks with Limited Data](https://mlsanity.com/paper/2006.06676/) (2020)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation](https://mlsanity.com/paper/1711.09020/) (2017)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [Interpreting the Latent Space of GANs for Semantic Face Editing](https://mlsanity.com/paper/1907.10786/) (2019)
- [CogView: Mastering Text-to-Image Generation via Transformers](https://mlsanity.com/paper/2105.13290/) (2021)
- [Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?](https://mlsanity.com/paper/1904.03189/) (2019)
- [GANSpace: Discovering Interpretable GAN Controls](https://mlsanity.com/paper/2004.02546/) (2020)
- [Generative Image Inpainting with Contextual Attention](https://mlsanity.com/paper/1801.07892/) (2018)
- [Designing an Encoder for StyleGAN Image Manipulation](https://mlsanity.com/paper/2102.02766/) (2021)
- [Image Inpainting for Irregular Holes Using Partial Convolutions](https://mlsanity.com/paper/1804.07723/) (2018)
- [Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors](https://mlsanity.com/paper/2203.13131/) (2022)
- [In-Domain GAN Inversion for Real Image Editing](https://mlsanity.com/paper/2004.00049/) (2020)
- [Image2StyleGAN++: How to Edit the Embedded Images?](https://mlsanity.com/paper/1911.11544/) (2019)

## Built on by (most cited first, 7 of 7)

- [DiffEdit: Diffusion-based semantic image editing with mask guidance](https://mlsanity.com/paper/2210.11427/) (2022)
- [TF-ICON: Diffusion-Based Training-Free Cross-Domain Image Composition](https://mlsanity.com/paper/2307.12493/) (2023)
- [GeneCIS: A Benchmark for General Conditional Image Similarity](https://mlsanity.com/paper/2306.07969/) (2023)
- [PrimeComposer: Faster Progressively Combined Diffusion for Image Composition with Attention Steering](https://mlsanity.com/paper/2403.05053/) (2024)
- [PAIR-Diffusion: A Comprehensive Multimodal Object-Level Image Editor](https://mlsanity.com/paper/2303.17546/) (2023)
- [Soundini: Sound-Guided Diffusion for Natural Video Editing](https://mlsanity.com/paper/2304.06818/) (2023)
- [FashionTex: Controllable Virtual Try-on with Text and Texture](https://mlsanity.com/paper/2305.04451/) (2023)

## Abstract (arXiv)

We consider the targeted image editing problem: blending a region in a source image with a driver image that specifies the desired change. Differently from prior works, we solve this problem by learning a conditional probability distribution of the edits, end-to-end. Training such a model requires addressing a fundamental technical challenge: the lack of example edits for training. To this end, we propose a self-supervised approach that simulates edits by augmenting off-the-shelf images in a target domain. The benefits are remarkable: implemented as a state-of-the-art auto-regressive transformer, our approach is simple, sidesteps difficulties with previous methods based on GAN-like priors, obtains significantly better edits, and is efficient. Furthermore, we show that different blending effects can be learned by an intuitive control of the augmentation process, with no other changes required to the model architecture. We demonstrate the superiority of this approach across several datasets in extensive quantitative and qualitative experiments, including human studies, significantly outperforming prior work.