# RetrieveGAN: Image Synthesis via Differentiable Patch Retrieval

arXiv: https://arxiv.org/abs/2007.08513 · HTML: https://mlsanity.com/paper/2007.08513/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jul 2020.

## Built on by (most cited first, 30 of 35)

- [Retrieval-Augmented Generation for AI-Generated Content: A Survey](https://mlsanity.com/paper/2402.19473/) (2024)
- [KNN-Diffusion: Image Generation via Large-Scale Retrieval](https://mlsanity.com/paper/2204.02849/) (2022)
- [Regularizing Generative Adversarial Networks under Limited Data](https://mlsanity.com/paper/2104.03310/) (2021)
- [A Review of the Gumbel-max Trick and its Extensions for Discrete Stochasticity in Machine Learning](https://mlsanity.com/paper/2110.01515/) (2021)
- [Make-A-Story: Visual Memory Conditioned Consistent Story Generation](https://mlsanity.com/paper/2211.13319/) (2022)
- [Enhancing Photorealism Enhancement](https://mlsanity.com/paper/2105.04619/) (2021)
- [Learning of Visual Relations: The Devil is in the Tails](https://mlsanity.com/paper/2108.09668/) (2021)
- [Text-Guided Synthesis of Artistic Images with Retrieval-Augmented Diffusion Models](https://mlsanity.com/paper/2207.13038/) (2022)
- [Improving Text-to-Image Synthesis Using Contrastive Learning](https://mlsanity.com/paper/2107.02423/) (2021)
- [In&Out : Diverse Image Outpainting via GAN Inversion](https://mlsanity.com/paper/2104.00675/) (2021)
- [Retrieval-based Controllable Molecule Generation](https://mlsanity.com/paper/2208.11126/) (2022)
- [Prompting Hard or Hardly Prompting: Prompt Inversion for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2312.12416/) (2023)
- [Semi-Parametric Neural Image Synthesis](https://mlsanity.com/paper/2204.11824/) (2022)
- [Exploiting Long-Term Dependencies for Generating Dynamic Scene Graphs](https://mlsanity.com/paper/2112.09828/) (2021)
- [Probabilistic Compositional Embeddings for Multimodal Image Retrieval](https://mlsanity.com/paper/2204.05845/) (2022)
- [RetrievalFuse: Neural 3D Scene Reconstruction with a Database](https://mlsanity.com/paper/2104.00024/) (2021)
- [Memories are One-to-Many Mapping Alleviators in Talking Face Generation](https://mlsanity.com/paper/2212.05005/) (2022)
- [Retrieval-Augmented Open-Vocabulary Object Detection](https://mlsanity.com/paper/2404.05687/) (2024)
- [Retrieval Based Time Series Forecasting](https://mlsanity.com/paper/2209.13525/) (2022)
- [Vector Quantized Image-to-Image Translation](https://mlsanity.com/paper/2207.13286/) (2022)
- [Retrieval-Augmented Score Distillation for Text-to-3D Generation](https://mlsanity.com/paper/2402.02972/) (2024)
- [Memory-Driven Text-to-Image Generation](https://mlsanity.com/paper/2208.07022/) (2022)
- [Nearest Neighbour Score Estimators for Diffusion Generative Models](https://mlsanity.com/paper/2402.08018/) (2024)
- [RAGDiffusion: Faithful Cloth Generation via External Knowledge Assimilation](https://mlsanity.com/paper/2411.19528/) (2024)
- [Decorating Your Own Bedroom: Locally Controlling Image Generation with Generative Adversarial Networks](https://mlsanity.com/paper/2105.08222/) (2021)
- [PatchRD: Detail-Preserving Shape Completion by Learning Patch Retrieval and Deformation](https://mlsanity.com/paper/2207.11790/) (2022)
- [Single-Stage Visual Relationship Learning using Conditional Queries](https://mlsanity.com/paper/2306.05689/) (2023)
- [Controllable Image Generation via Collage Representations](https://mlsanity.com/paper/2304.13722/) (2023)
- [RAGME: Retrieval Augmented Video Generation for Enhanced Motion Realism](https://mlsanity.com/paper/2504.06672/) (2025)
- [Remember What You have drawn: Semantic Image Manipulation with Memory](https://mlsanity.com/paper/2107.12579/) (2021)

## Abstract (arXiv)

Image generation from scene description is a cornerstone technique for the controlled generation, which is beneficial to applications such as content creation and image editing. In this work, we aim to synthesize images from scene description with retrieved patches as reference. We propose a differentiable retrieval module. With the differentiable retrieval module, we can (1) make the entire pipeline end-to-end trainable, enabling the learning of better feature embedding for retrieval; (2) encourage the selection of mutually compatible patches with additional objective functions. We conduct extensive quantitative and qualitative experiments to demonstrate that the proposed method can generate realistic and diverse images, where the retrieved patches are reasonable and mutually compatible.