# Generative Image Modeling using Style and Structure Adversarial Networks

arXiv: https://arxiv.org/abs/1603.05631 · HTML: https://mlsanity.com/paper/1603.05631/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Mar 2016.

## Built on by (most cited first, 30 of 50)

- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models](https://mlsanity.com/paper/1911.03393/) (2019)
- [GeoSim: Realistic Video Simulation via Geometry-Aware Composition for Self-Driving](https://mlsanity.com/paper/2101.06543/) (2021)
- [CAMPARI: Camera-Aware Decomposed Generative Neural Radiance Fields](https://mlsanity.com/paper/2103.17269/) (2021)
- [CommonScenes: Generating Commonsense 3D Indoor Scenes with Scene Graph Diffusion](https://mlsanity.com/paper/2305.16283/) (2023)
- [Exploring Frequency Adversarial Attacks for Face Forgery Detection](https://mlsanity.com/paper/2203.15674/) (2022)
- [Can segmentation models be trained with fully synthetically generated data?](https://mlsanity.com/paper/2209.08256/) (2022)
- [Video Autoencoder: self-supervised disentanglement of static 3D structure and motion](https://mlsanity.com/paper/2110.02951/) (2021)
- [Learned Spatial Representations for Few-shot Talking-Head Synthesis](https://mlsanity.com/paper/2104.14557/) (2021)
- [3D-Aware Indoor Scene Synthesis with Depth Priors](https://mlsanity.com/paper/2202.08553/) (2022)
- [3D-Aware Semantic-Guided Generative Model for Human Synthesis](https://mlsanity.com/paper/2112.01422/) (2021)
- [Compositional Transformers for Scene Generation](https://mlsanity.com/paper/2111.08960/) (2021)
- [CoordGAN: Self-Supervised Dense Correspondences Emerge from GANs](https://mlsanity.com/paper/2203.16521/) (2022)
- [A Survey on Deep Generative 3D-aware Image Synthesis](https://mlsanity.com/paper/2210.14267/) (2022)
- [AR-NeRF: Unsupervised Learning of Depth and Defocus Effects from Natural Images with Aperture Rendering Neural Radiance Fields](https://mlsanity.com/paper/2206.06100/) (2022)
- [A Diffusion Model Translator for Efficient Image-to-Image Translation](https://mlsanity.com/paper/2502.00307/) (2025)
- [Recurrent Affine Transformation for Text-to-image Synthesis](https://mlsanity.com/paper/2204.10482/) (2022)
- [MOGAN: Morphologic-structure-aware Generative Learning from a Single Image](https://mlsanity.com/paper/2103.02997/) (2021)
- [Partition-Guided GANs](https://mlsanity.com/paper/2104.00816/) (2021)
- [Unsupervised Novel View Synthesis from a Single Image](https://mlsanity.com/paper/2102.03285/) (2021)
- [Controllable Visual-Tactile Synthesis](https://mlsanity.com/paper/2305.03051/) (2023)
- [StEP: Style-based Encoder Pre-training for Multi-modal Image Synthesis](https://mlsanity.com/paper/2104.07098/) (2021)
- [Generator Knows What Discriminator Should Learn in Unconditional GANs](https://mlsanity.com/paper/2207.13320/) (2022)
- [LatentKeypointGAN: Controlling Images via Latent Keypoints](https://mlsanity.com/paper/2103.15812/) (2021)
- [Unsupervised Learning of Depth and Depth-of-Field Effect from Natural Images with Aperture Rendering Generative Adversarial Networks](https://mlsanity.com/paper/2106.13041/) (2021)
- [Scalable Font Reconstruction with Dual Latent Manifolds](https://mlsanity.com/paper/2109.06627/) (2021)
- [Generative Adversarial U-Net for Domain-free Medical Image Augmentation](https://mlsanity.com/paper/2101.04793/) (2021)
- [Consistent Multimodal Generation via A Unified GAN Framework](https://mlsanity.com/paper/2307.01425/) (2023)
- [PRISM: Progressive Restoration for Scene Graph-based Image Manipulation](https://mlsanity.com/paper/2311.02247/) (2023)
- [Pixel-to-Abundance Translation: Conditional Generative Adversarial Networks Based on Patch Transformer for Hyperspectral Unmixing](https://mlsanity.com/paper/2312.13127/) (2023)

## Abstract (arXiv)

Current generative frameworks use end-to-end learning and generate images by sampling from uniform noise distribution. However, these approaches ignore the most basic principle of image formation: images are product of: (a) Structure: the underlying 3D model; (b) Style: the texture mapped onto structure. In this paper, we factorize the image generation process and propose Style and Structure Generative Adversarial Network (S^2-GAN). Our S^2-GAN has two components: the Structure-GAN generates a surface normal map; the Style-GAN takes the surface normal map as input and generates the 2D image. Apart from a real vs. generated loss function, we use an additional loss with computed surface normals from generated images. The two GANs are first trained independently, and then merged together via joint learning. We show our S^2-GAN model is interpretable, generates more realistic images and can be used to learn unsupervised RGBD representations.