# Learning Generative Models of Textured 3D Meshes from Real-World Images

arXiv: https://arxiv.org/abs/2103.15627 · HTML: https://mlsanity.com/paper/2103.15627/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 58 references that are in the map.
- First posted on arXiv 29 Mar 2021.

## Builds on (most cited first, 30 of 48)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [A Point Set Generation Network for 3D Object Reconstruction from a Single Image](https://mlsanity.com/paper/1612.00603/) (2016)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [Self-Attention Generative Adversarial Networks](https://mlsanity.com/paper/1805.08318/) (2018)
- [GANSpace: Discovering Interpretable GAN Controls](https://mlsanity.com/paper/2004.02546/) (2020)
- [3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction](https://mlsanity.com/paper/1604.00449/) (2016)
- [Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling](https://mlsanity.com/paper/1610.07584/) (2016)
- [PointRend: Image Segmentation as Rendering](https://mlsanity.com/paper/1912.08193/) (2019)
- [Neural 3D Mesh Renderer](https://mlsanity.com/paper/1711.07566/) (2017)
- [Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning](https://mlsanity.com/paper/1904.01786/) (2019)
- [Learning Category-Specific Mesh Reconstruction from Image Collections](https://mlsanity.com/paper/1803.07549/) (2018)
- [cGANs with Projection Discriminator](https://mlsanity.com/paper/1802.05637/) (2018)
- [Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer](https://mlsanity.com/paper/1908.01210/) (2019)
- [Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs](https://mlsanity.com/paper/1703.09438/) (2017)
- [Multi-view Supervision for Single-view Reconstruction via Differentiable Ray Consistency](https://mlsanity.com/paper/1704.06254/) (2017)
- [Learning a Predictable and Generative Vector Representation for Objects](https://mlsanity.com/paper/1603.08637/) (2016)
- [Self-supervised Single-view 3D Reconstruction via Semantic Consistency](https://mlsanity.com/paper/2003.06473/) (2020)
- [Visual Object Networks: Image Generation with Disentangled 3D Representation](https://mlsanity.com/paper/1812.02725/) (2018)
- [MarrNet: 3D Shape Reconstruction via 2.5D Sketches](https://mlsanity.com/paper/1711.03129/) (2017)
- [Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision](https://mlsanity.com/paper/1612.00814/) (2016)
- [SCOPS: Self-Supervised Co-Part Segmentation](https://mlsanity.com/paper/1905.01298/) (2019)
- [Shape and Viewpoint without Keypoints](https://mlsanity.com/paper/2007.10982/) (2020)
- [Multiresolution Tree Networks for 3D Point Cloud Processing](https://mlsanity.com/paper/1807.03520/) (2018)

## Built on by (most cited first, 30 of 44)

- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [LION: Latent Point Diffusion Models for 3D Shape Generation](https://mlsanity.com/paper/2210.06978/) (2022)
- [GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images](https://mlsanity.com/paper/2209.11163/) (2022)
- [OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generation](https://mlsanity.com/paper/2301.07525/) (2023)
- [Dream3D: Zero-Shot Text-to-3D Synthesis Using 3D Shape Prior and Text-to-Image Diffusion Models](https://mlsanity.com/paper/2212.14704/) (2022)
- [MUSt3R: Multi-view Network for Stereo 3D Reconstruction](https://mlsanity.com/paper/2503.01661/) (2025)
- [4DGen: Grounded 4D Content Generation with Spatial-temporal Consistency](https://mlsanity.com/paper/2312.17225/) (2023)
- [EpiGRAF: Rethinking training of 3D GANs](https://mlsanity.com/paper/2206.10535/) (2022)
- [ATT3D: Amortized Text-to-3D Object Synthesis](https://mlsanity.com/paper/2306.07349/) (2023)
- [Texturify: Generating Textures on 3D Shape Surfaces](https://mlsanity.com/paper/2204.02411/) (2022)
- [Advances in 3D Generation: A Survey](https://mlsanity.com/paper/2401.17807/) (2024)
- [SPAD : Spatially Aware Multiview Diffusers](https://mlsanity.com/paper/2402.05235/) (2024)
- [FlashTex: Fast Relightable Mesh Texturing with LightControlNet](https://mlsanity.com/paper/2402.13251/) (2024)
- [TextureDreamer: Image-guided Texture Synthesis through Geometry-aware Diffusion](https://mlsanity.com/paper/2401.09416/) (2024)
- [Learning to Generate 3D Shapes from a Single Example](https://mlsanity.com/paper/2208.02946/) (2022)
- [Mesh2Tex: Generating Mesh Textures from Image Queries](https://mlsanity.com/paper/2304.05868/) (2023)
- [Shape, Pose, and Appearance from a Single Image via Bootstrapped Radiance Field Inversion](https://mlsanity.com/paper/2211.11674/) (2022)
- [Neural Assets: 3D-Aware Multi-Object Scene Synthesis with Image Diffusion Models](https://mlsanity.com/paper/2406.09292/) (2024)
- [Sin3DM: Learning a Diffusion Model from a Single 3D Textured Shape](https://mlsanity.com/paper/2305.15399/) (2023)
- [Controllable Text-to-3D Generation via Surface-Aligned Gaussian Splatting](https://mlsanity.com/paper/2403.09981/) (2024)
- [GINA-3D: Learning to Generate Implicit Neural Assets in the Wild](https://mlsanity.com/paper/2304.02163/) (2023)
- [PASTA: Controllable Part-Aware Shape Generation with Autoregressive Transformers](https://mlsanity.com/paper/2407.13677/) (2024)
- [ThemeStation: Generating Theme-Aware 3D Assets from Few Exemplars](https://mlsanity.com/paper/2403.15383/) (2024)
- [ShaDDR: Interactive Example-Based Geometry and Texture Generation via 3D Shape Detailization and Differentiable Rendering](https://mlsanity.com/paper/2306.04889/) (2023)
- [Pushing the Limits of 3D Shape Generation at Scale](https://mlsanity.com/paper/2306.11510/) (2023)
- [Single Mesh Diffusion Models with Field Latents for Texture Generation](https://mlsanity.com/paper/2312.09250/) (2023)
- [TUVF: Learning Generalizable Texture UV Radiance Fields](https://mlsanity.com/paper/2305.03040/) (2023)
- [SDFit: 3D Object Pose and Shape by Fitting a Morphable SDF to a Single Image](https://mlsanity.com/paper/2409.16178/) (2024)
- [MVControl: Adding Conditional Control to Multi-view Diffusion for Controllable Text-to-3D Generation](https://mlsanity.com/paper/2311.14494/) (2023)
- [ET3D: Efficient Text-to-3D Generation via Multi-View Distillation](https://mlsanity.com/paper/2311.15561/) (2023)

## Abstract (arXiv)

Recent advances in differentiable rendering have sparked an interest in learning generative models of textured 3D meshes from image collections. These models natively disentangle pose and appearance, enable downstream applications in computer graphics, and improve the ability of generative models to understand the concept of image formation. Although there has been prior work on learning such models from collections of 2D images, these approaches require a delicate pose estimation step that exploits annotated keypoints, thereby restricting their applicability to a few specific datasets. In this work, we propose a GAN framework for generating textured triangle meshes without relying on such annotations. We show that the performance of our approach is on par with prior work that relies on ground-truth keypoints, and more importantly, we demonstrate the generality of our method by setting new baselines on a larger set of categories from ImageNet - for which keypoints are not available - without any class-specific hyperparameter tuning. We release our code at https://github.com/dariopavllo/textured-3d-gan