# EpiGRAF: Rethinking training of 3D GANs

arXiv: https://arxiv.org/abs/2206.10535 · HTML: https://mlsanity.com/paper/2206.10535/

## Key facts

- Cited by 108 papers in the map (data through 14 Oct 2026).
- Builds on 67 of its 87 references that are in the map.
- First posted on arXiv 21 Jun 2022.

## Builds on (most cited first, 30 of 67)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [TensoRF: Tensorial Radiance Fields](https://mlsanity.com/paper/2203.09517/) (2022)
- [Efficient Geometry-aware 3D Generative Adversarial Networks](https://mlsanity.com/paper/2112.07945/) (2021)
- [Training Generative Adversarial Networks with Limited Data](https://mlsanity.com/paper/2006.06676/) (2020)
- [Learning Implicit Fields for Generative Shape Modeling](https://mlsanity.com/paper/1812.02822/) (2018)
- [StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation](https://mlsanity.com/paper/1711.09020/) (2017)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations](https://mlsanity.com/paper/1906.01618/) (2019)
- [Conditional Image Generation with PixelCNN Decoders](https://mlsanity.com/paper/1606.05328/) (2016)
- [NeRF++: Analyzing and Improving Neural Radiance Fields](https://mlsanity.com/paper/2010.07492/) (2020)
- [GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields](https://mlsanity.com/paper/2011.12100/) (2020)
- [Diffusion Probabilistic Models for 3D Point Cloud Generation](https://mlsanity.com/paper/2103.01458/) (2021)
- [pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2012.00926/) (2020)
- [Channel Pruning for Accelerating Very Deep Neural Networks](https://mlsanity.com/paper/1707.06168/) (2017)
- [Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision](https://mlsanity.com/paper/1912.07372/) (2019)
- [GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2007.02442/) (2020)
- [Which Training Methods for GANs do actually Converge?](https://mlsanity.com/paper/1801.04406/) (2018)

## Built on by (most cited first, 30 of 108)

- [Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model](https://mlsanity.com/paper/2311.06214/) (2023)
- [GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generation](https://mlsanity.com/paper/2403.14621/) (2024)
- [Generative Novel View Synthesis with 3D-Aware Diffusion Models](https://mlsanity.com/paper/2304.02602/) (2023)
- [DMV3D: Denoising Multi-View Diffusion using 3D Large Reconstruction Model](https://mlsanity.com/paper/2311.09217/) (2023)
- [Single-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstruction](https://mlsanity.com/paper/2304.06714/) (2023)
- [Video Probabilistic Diffusion Models in Projected Latent Space](https://mlsanity.com/paper/2302.07685/) (2023)
- [Next3D: Generative Neural Texture Rasterization for 3D-Aware Head Avatars](https://mlsanity.com/paper/2211.11208/) (2022)
- [EVA3D: Compositional 3D Human Generation from 2D Image Collections](https://mlsanity.com/paper/2210.04888/) (2022)
- [ZeroNVS: Zero-Shot 360-Degree View Synthesis from a Single Image](https://mlsanity.com/paper/2310.17994/) (2023)
- [HoloDiffusion: Training a 3D Diffusion Model using 2D Images](https://mlsanity.com/paper/2303.16509/) (2023)
- [Era3D: High-Resolution Multiview Diffusion using Efficient Row-wise Attention](https://mlsanity.com/paper/2405.11616/) (2024)
- [PanoHead: Geometry-Aware 3D Full-Head Synthesis in 360circ](https://mlsanity.com/paper/2303.13071/) (2023)
- [SceneDreamer: Unbounded 3D Scene Generation from 2D Image Collections](https://mlsanity.com/paper/2302.01330/) (2023)
- [3D-aware Image Generation using 2D Diffusion Models](https://mlsanity.com/paper/2303.17905/) (2023)
- [InfiniCity: Infinite-Scale City Synthesis](https://mlsanity.com/paper/2301.09637/) (2023)
- [Real-Time Radiance Fields for Single-Image Portrait View Synthesis](https://mlsanity.com/paper/2305.02310/) (2023)
- [Texture Generation on 3D Meshes with Point-UV Diffusion](https://mlsanity.com/paper/2308.10490/) (2023)
- [SPAD : Spatially Aware Multiview Diffusers](https://mlsanity.com/paper/2402.05235/) (2024)
- [DisCoScene: Spatially Disentangled Generative Radiance Fields for Controllable 3D-aware Scene Synthesis](https://mlsanity.com/paper/2212.11984/) (2022)
- [CC3D: Layout-Conditioned Generation of Compositional 3D Scenes](https://mlsanity.com/paper/2303.12074/) (2023)
- [GGHead: Fast and Generalizable 3D Gaussian Heads](https://mlsanity.com/paper/2406.09377/) (2024)
- [AutoDecoding Latent 3D Diffusion Models](https://mlsanity.com/paper/2307.05445/) (2023)
- [3D generation on ImageNet](https://mlsanity.com/paper/2303.01416/) (2023)
- [3D GAN Inversion with Pose Optimization](https://mlsanity.com/paper/2210.07301/) (2022)
- [Persistent Nature: A Generative Model of Unbounded 3D Worlds](https://mlsanity.com/paper/2303.13515/) (2023)
- [L3DG: Latent 3D Gaussian Diffusion](https://mlsanity.com/paper/2410.13530/) (2024)
- [GeoLRM: Geometry-Aware Large Reconstruction Model for High-Quality 3D Gaussian Generation](https://mlsanity.com/paper/2406.15333/) (2024)
- [Make Encoder Great Again in 3D GAN Inversion through Geometry and Occlusion-Aware Encoding](https://mlsanity.com/paper/2303.12326/) (2023)
- [OmniAvatar: Geometry-Guided Controllable 3D Head Synthesis](https://mlsanity.com/paper/2303.15539/) (2023)
- [VOODOO 3D: Volumetric Portrait Disentanglement for One-Shot 3D Head Reenactment](https://mlsanity.com/paper/2312.04651/) (2023)

## Abstract (arXiv)

A very recent trend in generative modeling is building 3D-aware generators from 2D image collections. To induce the 3D bias, such models typically rely on volumetric rendering, which is expensive to employ at high resolutions. During the past months, there appeared more than 10 works that address this scaling issue by training a separate 2D decoder to upsample a low-resolution image (or a feature tensor) produced from a pure 3D generator. But this solution comes at a cost: not only does it break multi-view consistency (i.e. shape and texture change when the camera moves), but it also learns the geometry in a low fidelity. In this work, we show that it is possible to obtain a high-resolution 3D generator with SotA image quality by following a completely different route of simply training the model patch-wise. We revisit and improve this optimization scheme in two ways. First, we design a location- and scale-aware discriminator to work on patches of different proportions and spatial positions. Second, we modify the patch sampling strategy based on an annealed beta distribution to stabilize training and accelerate the convergence. The resulted model, named EpiGRAF, is an efficient, high-resolution, pure 3D generator, and we test it on four datasets (two introduced in this work) at 2562 and 5122 resolutions. It obtains state-of-the-art image quality, high-fidelity geometry and trains ≈ 2.5 × faster than the upsampler-based counterparts. Project website: https://universome.github.io/epigraf.