# Style Agnostic 3D Reconstruction via Adversarial Style Transfer

arXiv: https://arxiv.org/abs/2110.10784 · HTML: https://mlsanity.com/paper/2110.10784/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 34 references that are in the map.
- First posted on arXiv 20 Oct 2021.

## Builds on (most cited first, 24 of 24)

- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Adversarial Discriminative Domain Adaptation](https://mlsanity.com/paper/1702.05464/) (2017)
- [CyCADA: Cycle-Consistent Adversarial Domain Adaptation](https://mlsanity.com/paper/1711.03213/) (2017)
- [A Point Set Generation Network for 3D Object Reconstruction from a Single Image](https://mlsanity.com/paper/1612.00603/) (2016)
- [Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images](https://mlsanity.com/paper/1804.01654/) (2018)
- [3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction](https://mlsanity.com/paper/1604.00449/) (2016)
- [A Neural Algorithm of Artistic Style](https://mlsanity.com/paper/1508.06576/) (2015)
- [Neural 3D Mesh Renderer](https://mlsanity.com/paper/1711.07566/) (2017)
- [Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning](https://mlsanity.com/paper/1904.01786/) (2019)
- [A Survey of Unsupervised Deep Domain Adaptation](https://mlsanity.com/paper/1812.02849/) (2018)
- [Learning Category-Specific Mesh Reconstruction from Image Collections](https://mlsanity.com/paper/1803.07549/) (2018)
- [Self-supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2004.04581/) (2020)
- [Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer](https://mlsanity.com/paper/1908.01210/) (2019)
- [SDFDiff: Differentiable Rendering of Signed Distance Fields for 3D Shape Optimization](https://mlsanity.com/paper/1912.07109/) (2019)
- [Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision](https://mlsanity.com/paper/1612.00814/) (2016)
- [Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Rendering](https://mlsanity.com/paper/2010.09125/) (2020)
- [Learning Detailed Face Reconstruction from a Single Image](https://mlsanity.com/paper/1611.05053/) (2016)
- [Leveraging 2D Data to Learn Textured 3D Mesh Generation](https://mlsanity.com/paper/2004.04180/) (2020)
- [CycleGAN, a Master of Steganography](https://mlsanity.com/paper/1712.02950/) (2017)
- [LIME: Live Intrinsic Material Estimation](https://mlsanity.com/paper/1801.01075/) (2018)
- [DeformNet: Free-Form Deformation Network for 3D Shape Reconstruction from a Single Image](https://mlsanity.com/paper/1708.04672/) (2017)
- [Learning to Generate and Reconstruct 3D Meshes with only 2D Supervision](https://mlsanity.com/paper/1807.09259/) (2018)
- [Single-View 3D Object Reconstruction from Shape Priors in Memory](https://mlsanity.com/paper/2003.03711/) (2020)

## Built on by (most cited first, 3 of 3)

- [GenDR: A Generalized Differentiable Renderer](https://mlsanity.com/paper/2204.13845/) (2022)
- [Generalizing Stochastic Smoothing for Differentiation and Gradient Estimation](https://mlsanity.com/paper/2410.08125/) (2024)
- [Newton Losses: Using Curvature Information for Learning with Differentiable Algorithms](https://mlsanity.com/paper/2410.19055/) (2024)

## Abstract (arXiv)

Reconstructing the 3D geometry of an object from an image is a major challenge in computer vision. Recently introduced differentiable renderers can be leveraged to learn the 3D geometry of objects from 2D images, but those approaches require additional supervision to enable the renderer to produce an output that can be compared to the input image. This can be scene information or constraints such as object silhouettes, uniform backgrounds, material, texture, and lighting. In this paper, we propose an approach that enables a differentiable rendering-based learning of 3D objects from images with backgrounds without the need for silhouette supervision. Instead of trying to render an image close to the input, we propose an adversarial style-transfer and domain adaptation pipeline that allows to translate the input image domain to the rendered image domain. This allows us to directly compare between a translated image and the differentiable rendering of a 3D object reconstruction in order to train the 3D object reconstruction network. We show that the approach learns 3D geometry from images with backgrounds and provides a better performance than constrained methods for single-view 3D object reconstruction on this task.