# Semantic Pyramid for Image Generation

arXiv: https://arxiv.org/abs/2003.06221 · HTML: https://mlsanity.com/paper/2003.06221/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Mar 2020.

## Built on by (most cited first, 30 of 34)

- [ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2108.02938/) (2021)
- [Paint by Example: Exemplar-based Image Editing with Diffusion Models](https://mlsanity.com/paper/2211.13227/) (2022)
- [Deep ViT Features as Dense Visual Descriptors](https://mlsanity.com/paper/2112.05814/) (2021)
- [VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance](https://mlsanity.com/paper/2204.08583/) (2022)
- [Projected GANs Converge Faster](https://mlsanity.com/paper/2111.01007/) (2021)
- [Explaining in Style: Training a GAN to explain a classifier in StyleSpace](https://mlsanity.com/paper/2104.13369/) (2021)
- [Image-to-Image Translation: Methods and Applications](https://mlsanity.com/paper/2101.08629/) (2021)
- [Exploiting Spatial Dimensions of Latent in GAN for Real-time Image Editing](https://mlsanity.com/paper/2104.14754/) (2021)
- [Ensembling Off-the-shelf Models for GAN Training](https://mlsanity.com/paper/2112.09130/) (2021)
- [High Fidelity Visualization of What Your Self-Supervised Representation Knows About](https://mlsanity.com/paper/2112.09164/) (2021)
- [Using latent space regression to analyze and leverage compositionality in GANs](https://mlsanity.com/paper/2103.10426/) (2021)
- [Rosetta Neurons: Mining the Common Units in a Model Zoo](https://mlsanity.com/paper/2306.09346/) (2023)
- [Pose with Style: Detail-Preserving Pose-Guided Image Synthesis with Conditional StyleGAN](https://mlsanity.com/paper/2109.06166/) (2021)
- [SceneComposer: Any-Level Semantic Image Synthesis](https://mlsanity.com/paper/2211.11742/) (2022)
- [Guided Co-Modulated GAN for 360deg Field of View Extrapolation](https://mlsanity.com/paper/2204.07286/) (2022)
- [IMAGINE: Image Synthesis by Image-Guided Model Inversion](https://mlsanity.com/paper/2104.05895/) (2021)
- [Wireless End-to-End Image Transmission System using Semantic Communications](https://mlsanity.com/paper/2302.13721/) (2023)
- [Facial-Sketch Synthesis: A New Challenge](https://mlsanity.com/paper/2112.15439/) (2021)
- [Neural Semantic Surface Maps](https://mlsanity.com/paper/2309.04836/) (2023)
- [Contrastive Feature Loss for Image Prediction](https://mlsanity.com/paper/2111.06934/) (2021)
- [Understanding Invariance via Feedforward Inversion of Discriminatively Trained Classifiers](https://mlsanity.com/paper/2103.07470/) (2021)
- [MOGAN: Morphologic-structure-aware Generative Learning from a Single Image](https://mlsanity.com/paper/2103.02997/) (2021)
- [Hierarchical Semantic Regularization of Latent Spaces in StyleGANs](https://mlsanity.com/paper/2208.03764/) (2022)
- [TransferI2I: Transfer Learning for Image-to-Image Translation from Small Datasets](https://mlsanity.com/paper/2105.06219/) (2021)
- [End-to-End Visual Editing with a Generatively Pre-Trained Artist](https://mlsanity.com/paper/2205.01668/) (2022)
- [Generator Knows What Discriminator Should Learn in Unconditional GANs](https://mlsanity.com/paper/2207.13320/) (2022)
- [NaturalInversion: Data-Free Image Synthesis Improving Real-World Consistency](https://mlsanity.com/paper/2306.16661/) (2023)
- [Inverting Adversarially Robust Networks for Image Synthesis](https://mlsanity.com/paper/2106.06927/) (2021)
- [Controllable Image Generation via Collage Representations](https://mlsanity.com/paper/2304.13722/) (2023)
- [DDR: Exploiting Deep Degradation Response as Flexible Image Descriptor](https://mlsanity.com/paper/2406.08377/) (2024)

## Abstract (arXiv)

We present a novel GAN-based model that utilizes the space of deep features learned by a pre-trained classification model. Inspired by classical image pyramid representations, we construct our model as a Semantic Generation Pyramid -- a hierarchical framework which leverages the continuum of semantic information encapsulated in such deep features; this ranges from low level information contained in fine features to high level, semantic information contained in deeper features. More specifically, given a set of features extracted from a reference image, our model generates diverse image samples, each with matching features at each semantic level of the classification model. We demonstrate that our model results in a versatile and flexible framework that can be used in various classic and novel image generation tasks. These include: generating images with a controllable extent of semantic similarity to a reference image, and different manipulation tasks such as semantically-controlled inpainting and compositing; all achieved with the same model, with no further training.