# Drop the GAN: In Defense of Patches Nearest Neighbors as Single Image Generative Models

arXiv: https://arxiv.org/abs/2103.15545 · HTML: https://mlsanity.com/paper/2103.15545/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 43 references that are in the map.
- First posted on arXiv 29 Mar 2021.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Deep Image Prior](https://mlsanity.com/paper/1711.10925/) (2017)
- [Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks](https://mlsanity.com/paper/1703.10593/) (2017)
- [SinGAN: Learning a Generative Model from a Single Natural Image](https://mlsanity.com/paper/1905.01164/) (2019)
- ["Zero-Shot" Super-Resolution using Deep Internal Learning](https://mlsanity.com/paper/1712.06087/) (2017)
- [Improved Texture Networks: Maximizing Quality and Diversity in Feed-forward Stylization and Texture Synthesis](https://mlsanity.com/paper/1701.02096/) (2017)
- [Style Transfer by Relaxed Optimal Transport and Self-Similarity](https://mlsanity.com/paper/1904.12785/) (2019)
- [Visual Attribute Transfer through Deep Image Analogy](https://mlsanity.com/paper/1705.01088/) (2017)
- ["Double-DIP": Unsupervised Image Decomposition via Coupled Deep-Image-Priors](https://mlsanity.com/paper/1812.00467/) (2018)
- [Improved Techniques for Training Single-Image GANs](https://mlsanity.com/paper/2003.11512/) (2020)
- [Places: An Image Database for Deep Scene Understanding](https://mlsanity.com/paper/1610.02055/) (2016)
- [Positional Encoding as Spatial Inductive Bias in GANs](https://mlsanity.com/paper/2012.05217/) (2020)
- [Non-Stationary Texture Synthesis by Adversarial Expansion](https://mlsanity.com/paper/1805.04487/) (2018)
- [Learning Texture Manifolds with the Periodic Spatial GAN](https://mlsanity.com/paper/1705.06566/) (2017)
- [Texture Synthesis with Spatial Generative Adversarial Networks](https://mlsanity.com/paper/1611.08207/) (2016)
- [Hierarchical Patch VAE-GAN: Generating Diverse Videos from a Single Sample](https://mlsanity.com/paper/2006.12226/) (2020)
- [An Internal Learning Approach to Video Inpainting](https://mlsanity.com/paper/1909.07957/) (2019)
- [TuiGAN: Learning Versatile Image-to-Image Translation with Two Unpaired Images](https://mlsanity.com/paper/2004.04634/) (2020)
- [Style-Transfer via Texture-Synthesis](https://mlsanity.com/paper/1609.03057/) (2016)
- [Structural-analogy from a Single Image Pair](https://mlsanity.com/paper/2004.02222/) (2020)
- [DCIL: Deep Contextual Internal Learning for Image Restoration and Image Retargeting](https://mlsanity.com/paper/1912.04229/) (2019)
- [MOGAN: Morphologic-structure-aware Generative Learning from a Single Image](https://mlsanity.com/paper/2103.02997/) (2021)
- [DeepCFL: Deep Contextual Features Learning from a Single Image](https://mlsanity.com/paper/2011.03712/) (2020)
- [Example-Based Image Synthesis via Randomized Patch-Matching](https://mlsanity.com/paper/1609.07370/) (2016)

## Built on by (most cited first, 30 of 51)

- [SINE: SINgle Image Editing with Text-to-Image Diffusion Models](https://mlsanity.com/paper/2212.04489/) (2022)
- [Diffusion-based Image Translation using Disentangled Style and Content Representation](https://mlsanity.com/paper/2209.15264/) (2022)
- [SPIn-NeRF: Multiview Segmentation and Perceptual Inpainting with Neural Radiance Fields](https://mlsanity.com/paper/2211.12254/) (2022)
- [Single Motion Diffusion](https://mlsanity.com/paper/2302.05905/) (2023)
- [SinDDM: A Single Image Denoising Diffusion Model](https://mlsanity.com/paper/2211.16582/) (2022)
- [SinFusion: Training Diffusion Models on a Single Image or Video](https://mlsanity.com/paper/2211.11743/) (2022)
- [SinDiffusion: Learning a Diffusion Model from a Single Natural Image](https://mlsanity.com/paper/2211.12445/) (2022)
- [Self-Distilled StyleGAN: Towards Generation from Internet Photos](https://mlsanity.com/paper/2202.12211/) (2022)
- [Learning to Generate 3D Shapes from a Single Example](https://mlsanity.com/paper/2208.02946/) (2022)
- [Green Learning: Introduction, Examples and Outlook](https://mlsanity.com/paper/2210.00965/) (2022)
- [Sin3DM: Learning a Diffusion Model from a Single 3D Textured Shape](https://mlsanity.com/paper/2305.15399/) (2023)
- [Generating natural images with direct Patch Distributions Matching](https://mlsanity.com/paper/2203.11862/) (2022)
- [Hierarchical Patch Diffusion Models for High-Resolution Video Generation](https://mlsanity.com/paper/2406.07792/) (2024)
- [PatchNR: Learning from Very Few Images by Patch Normalizing Flow Regularization](https://mlsanity.com/paper/2205.12021/) (2022)
- [Rethinking Exemplars for Continual Semantic Segmentation in Endoscopy Scenes: Entropy-based Mini-Batch Pseudo-Replay](https://mlsanity.com/paper/2308.14100/) (2023)
- [Example-based Motion Synthesis via Generative Motion Matching](https://mlsanity.com/paper/2306.00378/) (2023)
- [Patch-based 3D Natural Scene Generation from a Single Example](https://mlsanity.com/paper/2304.12670/) (2023)
- [PetsGAN: Rethinking Priors for Single Image Generation](https://mlsanity.com/paper/2203.01488/) (2022)
- [Diverse Generation from a Single Video Made Possible](https://mlsanity.com/paper/2109.08591/) (2021)
- [Seeing a Rose in Five Thousand Ways](https://mlsanity.com/paper/2212.04965/) (2022)
- [ExSinGAN: Learning an Explainable Generative Model from a Single Image](https://mlsanity.com/paper/2105.07350/) (2021)
- [MOPA: Modular Object Navigation with PointGoal Agents](https://mlsanity.com/paper/2304.03696/) (2023)
- [Single Mesh Diffusion Models with Field Latents for Texture Generation](https://mlsanity.com/paper/2312.09250/) (2023)
- [Image Synthesis under Limited Data: A Survey and Taxonomy](https://mlsanity.com/paper/2307.16879/) (2023)
- [Learning from small data sets: Patch-based regularizers in inverse problems for image reconstruction](https://mlsanity.com/paper/2312.16611/) (2023)
- [MotionDreamer: One-to-Many Motion Synthesis with Localized Generative Masked Transformer](https://mlsanity.com/paper/2504.08959/) (2025)
- [Rethinking Nearest Neighbors for Visual Classification](https://mlsanity.com/paper/2112.08459/) (2021)
- [MAGIC: Mask-Guided Image Synthesis by Inverting a Quasi-Robust Classifier](https://mlsanity.com/paper/2209.11549/) (2022)
- [3D Video Loops from Asynchronous Input](https://mlsanity.com/paper/2303.05312/) (2023)
- [FIANCEE: Faster Inference of Adversarial Networks via Conditional Early Exits](https://mlsanity.com/paper/2304.10306/) (2023)

## Abstract (arXiv)

Single image generative models perform synthesis and manipulation tasks by capturing the distribution of patches within a single image. The classical (pre Deep Learning) prevailing approaches for these tasks are based on an optimization process that maximizes patch similarity between the input and generated output. Recently, however, Single Image GANs were introduced both as a superior solution for such manipulation tasks, but also for remarkable novel generative tasks. Despite their impressiveness, single image GANs require long training time (usually hours) for each image and each task. They often suffer from artifacts and are prone to optimization issues such as mode collapse. In this paper, we show that all of these tasks can be performed without any training, within several seconds, in a unified, surprisingly simple framework. We revisit and cast the "good-old" patch-based methods into a novel optimization-free framework. We start with an initial coarse guess, and then simply refine the details coarse-to-fine using patch-nearest-neighbor search. This allows generating random novel images better and much faster than GANs. We further demonstrate a wide range of applications, such as image editing and reshuffling, retargeting to different sizes, structural analogies, image collage and a newly introduced task of conditional inpainting. Not only is our method faster (× 103-× 104 than a GAN), it produces superior results (confirmed by quantitative and qualitative evaluation), less artifacts and more realistic global structure than any of the previous approaches (whether GAN-based or classical patch-based).