# Generating Novel Scene Compositions from Single Images and Videos

arXiv: https://arxiv.org/abs/2103.13389 · HTML: https://mlsanity.com/paper/2103.13389/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 76 references that are in the map.
- First posted on arXiv 24 Mar 2021.

## Builds on (most cited first, 30 of 50)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Training Generative Adversarial Networks with Limited Data](https://mlsanity.com/paper/2006.06676/) (2020)
- [Deep Image Prior](https://mlsanity.com/paper/1711.10925/) (2017)
- [StarGAN v2: Diverse Image Synthesis for Multiple Domains](https://mlsanity.com/paper/1912.01865/) (2019)
- [Self-Attention Generative Adversarial Networks](https://mlsanity.com/paper/1805.08318/) (2018)
- [Multimodal Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1804.04732/) (2018)
- [Differentiable Augmentation for Data-Efficient GAN Training](https://mlsanity.com/paper/2006.10738/) (2020)
- [SinGAN: Learning a Generative Model from a Single Natural Image](https://mlsanity.com/paper/1905.01164/) (2019)
- ["Zero-Shot" Super-Resolution using Deep Internal Learning](https://mlsanity.com/paper/1712.06087/) (2017)
- [Data Augmentation Generative Adversarial Networks](https://mlsanity.com/paper/1711.04340/) (2017)
- [YFCC100M: The New Data in Multimedia Research](https://mlsanity.com/paper/1503.01817/) (2015)
- [Generating Images with Perceptual Similarity Metrics based on Deep Networks](https://mlsanity.com/paper/1602.02644/) (2016)
- [Improved Texture Networks: Maximizing Quality and Diversity in Feed-forward Stylization and Texture Synthesis](https://mlsanity.com/paper/1701.02096/) (2017)
- [Swapping Autoencoder for Deep Image Manipulation](https://mlsanity.com/paper/2007.00653/) (2020)
- [SRFlow: Learning the Super-Resolution Space with Normalizing Flow](https://mlsanity.com/paper/2006.14200/) (2020)
- [Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks](https://mlsanity.com/paper/1604.04382/) (2016)
- [Blind Super-Resolution Kernel Estimation using an Internal-GAN](https://mlsanity.com/paper/1909.06581/) (2019)
- [Texture Networks: Feed-forward Synthesis of Textures and Stylized Images](https://mlsanity.com/paper/1603.03417/) (2016)
- [BAM: Bottleneck Attention Module](https://mlsanity.com/paper/1807.06514/) (2018)
- [Towards Faster and Stabilized GAN Training for High-fidelity Few-shot Image Synthesis](https://mlsanity.com/paper/2101.04775/) (2021)

## Built on by (most cited first, 8 of 8)

- [Single Motion Diffusion](https://mlsanity.com/paper/2302.05905/) (2023)
- [SinDiffusion: Learning a Diffusion Model from a Single Natural Image](https://mlsanity.com/paper/2211.12445/) (2022)
- [Generating natural images with direct Patch Distributions Matching](https://mlsanity.com/paper/2203.11862/) (2022)
- [FreGAN: Exploiting Frequency Components for Training GANs under Limited Data](https://mlsanity.com/paper/2210.05461/) (2022)
- [Comprehensive Review of EEG-to-Output Research: Decoding Neural Signals into Images, Videos, and Audio](https://mlsanity.com/paper/2412.19999/) (2024)
- [One-shot Ultra-high-Resolution Generative Adversarial Network That Synthesizes 16K Images On A Single GPU](https://mlsanity.com/paper/2202.13799/) (2022)
- [VoloGAN: Adversarial Domain Adaptation for Synthetic Depth Data](https://mlsanity.com/paper/2207.09204/) (2022)
- [Conditioning GAN Without Training Dataset](https://mlsanity.com/paper/2405.20687/) (2024)

## Abstract (arXiv)

Given a large dataset for training, generative adversarial networks (GANs) can achieve remarkable performance for the image synthesis task. However, training GANs in extremely low data regimes remains a challenge, as overfitting often occurs, leading to memorization or training divergence. In this work, we introduce SIV-GAN, an unconditional generative model that can generate new scene compositions from a single training image or a single video clip. We propose a two-branch discriminator architecture, with content and layout branches designed to judge internal content and scene layout realism separately from each other. This discriminator design enables synthesis of visually plausible, novel compositions of a scene, with varying content and layout, while preserving the context of the original sample. Compared to previous single image GANs, our model generates more diverse, higher quality images, while not being restricted to a single image setting. We further introduce a new challenging task of learning from a few frames of a single video. In this training setup the training images are highly similar to each other, which makes it difficult for prior GAN models to achieve a synthesis of both high quality and diversity.