# Visual Object Networks: Image Generation with Disentangled 3D Representation

arXiv: https://arxiv.org/abs/1812.02725 · HTML: https://mlsanity.com/paper/1812.02725/

## Key facts

- Cited by 124 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Dec 2018.

## Built on by (most cited first, 30 of 124)

- [Efficient Geometry-aware 3D Generative Adversarial Networks](https://mlsanity.com/paper/2112.07945/) (2021)
- [Structured 3D Latents for Scalable and Versatile 3D Generation](https://mlsanity.com/paper/2412.01506/) (2024)
- [Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering](https://mlsanity.com/paper/2106.02634/) (2021)
- [ReconFusion: 3D Reconstruction with Diffusion Priors](https://mlsanity.com/paper/2312.02981/) (2023)
- [Editing Conditional Radiance Fields](https://mlsanity.com/paper/2105.06466/) (2021)
- [GRAM: Generative Radiance Manifolds for 3D-Aware Image Generation](https://mlsanity.com/paper/2112.08867/) (2021)
- [CIPS-3D: A 3D-Aware Generator of GANs Based on Conditionally-Independent Pixel Synthesis](https://mlsanity.com/paper/2110.09788/) (2021)
- [VoxGRAF: Fast 3D-Aware Image Synthesis with Sparse Voxel Grids](https://mlsanity.com/paper/2206.07695/) (2022)
- [Next3D: Generative Neural Texture Rasterization for 3D-Aware Head Avatars](https://mlsanity.com/paper/2211.11208/) (2022)
- [A-SDF: Learning Disentangled Signed Distance Functions for Articulated Shape Representation](https://mlsanity.com/paper/2104.07645/) (2021)
- [FENeRF: Face Editing in Neural Radiance Fields](https://mlsanity.com/paper/2111.15490/) (2021)
- [Disentangled Representation Learning](https://mlsanity.com/paper/2211.11695/) (2022)
- [3D Neural Scene Representations for Visuomotor Control](https://mlsanity.com/paper/2107.04004/) (2021)
- [3D-aware Image Synthesis via Learning Structural and Textural Representations](https://mlsanity.com/paper/2112.10759/) (2021)
- [Generative Neural Articulated Radiance Fields](https://mlsanity.com/paper/2206.14314/) (2022)
- [V3D: Video Diffusion Models are Effective 3D Generators](https://mlsanity.com/paper/2403.06738/) (2024)
- [GeoSim: Realistic Video Simulation via Geometry-Aware Composition for Self-Driving](https://mlsanity.com/paper/2101.06543/) (2021)
- [A Shading-Guided Generative Implicit Model for Shape-Accurate 3D-Aware Image Synthesis](https://mlsanity.com/paper/2110.15678/) (2021)
- [Texturify: Generating Textures on 3D Shape Surfaces](https://mlsanity.com/paper/2204.02411/) (2022)
- [Unsupervised Learning of Efficient Geometry-Aware Neural Articulated Representations](https://mlsanity.com/paper/2204.08839/) (2022)
- [Advances in 3D Generation: A Survey](https://mlsanity.com/paper/2401.17807/) (2024)
- [DisCoScene: Spatially Disentangled Generative Radiance Fields for Controllable 3D-aware Scene Synthesis](https://mlsanity.com/paper/2212.11984/) (2022)
- [Neural Re-Rendering of Humans from a Single Image](https://mlsanity.com/paper/2101.04104/) (2021)
- [Control-NeRF: Editable Feature Volumes for Scene Rendering and Manipulation](https://mlsanity.com/paper/2204.10850/) (2022)
- [Generative Multiplane Images: Making a 2D GAN 3D-Aware](https://mlsanity.com/paper/2207.10642/) (2022)
- [CAMPARI: Camera-Aware Decomposed Generative Neural Radiance Fields](https://mlsanity.com/paper/2103.17269/) (2021)
- [Amodal3R: Amodal 3D Reconstruction from Occluded 2D Images](https://mlsanity.com/paper/2503.13439/) (2025)
- [3D GAN Inversion for Controllable Portrait Image Animation](https://mlsanity.com/paper/2203.13441/) (2022)
- [Multimodal Image Synthesis and Editing: The Generative AI Era](https://mlsanity.com/paper/2112.13592/) (2021)
- [Generative Occupancy Fields for 3D Surface-Aware Image Synthesis](https://mlsanity.com/paper/2111.00969/) (2021)

## Abstract (arXiv)

Recent progress in deep generative models has led to tremendous breakthroughs in image generation. However, while existing models can synthesize photorealistic images, they lack an understanding of our underlying 3D world. We present a new generative model, Visual Object Networks (VON), synthesizing natural images of objects with a disentangled 3D representation. Inspired by classic graphics rendering pipelines, we unravel our image formation process into three conditionally independent factors---shape, viewpoint, and texture---and present an end-to-end adversarial learning framework that jointly models 3D shapes and 2D images. Our model first learns to synthesize 3D shapes that are indistinguishable from real shapes. It then renders the object's 2.5D sketches (i.e., silhouette and depth map) from its shape under a sampled viewpoint. Finally, it learns to add realistic texture to these 2.5D sketches to generate natural images. The VON not only generates images that are more realistic than state-of-the-art 2D image synthesis methods, but also enables many 3D operations such as changing the viewpoint of a generated image, editing of shape and texture, linear interpolation in texture and shape space, and transferring appearance across different objects and viewpoints.