mlsanityData to 14 Oct 2026

GaussiGAN: Controllable Image Synthesis with 3D Gaussians from Unposed Silhouettes

arXiv 2106.13215 ↗First posted 24 Jun 2021cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

We present an algorithm that learns a coarse 3D representation of objects from unposed multi-view 2D mask supervision, then uses it to generate detailed mask and image texture. In contrast to existing voxel-based methods for unposed object reconstruction, our approach learns to represent the generated shape and pose with a set of self-supervised canonical 3D anisotropic Gaussians via a perspective camera, and a set of per-image transforms. We show that this approach can robustly estimate a 3D space for the camera and object, while recent baselines sometimes struggle to reconstruct coherent 3D spaces in this setting. We show results on synthetic datasets with realistic lighting, and demonstrate object insertion with interactive posing. With our work, we help move towards structured representations that handle more real-world variation in learning-based object reconstruction.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 49 of its 64 references that are in the map. First posted on arXiv 24 Jun 2021.

Builds on

49 of 64 references.

References · 49

  1. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  2. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  3. 2013Auto-Encoding Variational Bayescited by 4,053
  4. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  5. 2016Improved Techniques for Training GANscited by 2,970
  6. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,694
  7. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,636
  8. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,299
  9. 2016Stacked Hourglass Networks for Human Pose Estimationcited by 915
  10. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 785
  11. 2017Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networkscited by 707
  12. 2020GRAF: Generative Radiance Fields for 3D-Aware Image Synthesiscited by 689
37 more
  1. 2019PointRend: Image Segmentation as Renderingcited by 440
  2. 2018DeepVoxels: Learning Persistent 3D Feature Embeddingscited by 405
  3. 2018Learning Category-Specific Mesh Reconstruction from Image Collectionscited by 317
  4. 2017Toward Multimodal Image-to-Image Translationcited by 296
  5. 2020StyleRig: Rigging StyleGAN for 3D Control over Portrait Imagescited by 268
  6. 2019Learning Shape Templates with Structured Implicit Functionscited by 242
  7. 2017Geometric GANcited by 233
  8. 2018Animating Arbitrary Objects via Deep Motion Transfercited by 208
  9. 2016Learning Shape Abstractions by Assembling Volumetric Primitivescited by 181
  10. 2019CvxNet: Learnable Convex Decompositioncited by 174
  11. 2018Escaping Plato's Cave: 3D Shape From Adversarial Renderingcited by 170
  12. 2020BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Imagescited by 166
  13. 2017Deep Image Harmonizationcited by 120
  14. 2018Unsupervised Learning of Object Landmarks through Conditional Image Generationcited by 114
  15. 2019Three-D Safari: Learning to Estimate Zebra Pose, Shape, and Texture from Images "In the Wild"cited by 106
  16. 2019Towards Unsupervised Learning of Generative Models for 3D Controllable Image Synthesiscited by 104
  17. 2018ST-GAN: Spatial Transformer Generative Adversarial Networks for Image Compositingcited by 90
  18. 2020Articulation-aware Canonical Surface Mappingcited by 81
  19. 2019Unsupervised Part-Based Disentangling of Object Shape and Appearancecited by 67
  20. 2017Unsupervised learning of object landmarks by factorized spatial embeddingscited by 64
  21. 2018Im2Struct: Recovering 3D Shape Structure from a Single RGB Imagecited by 55
  22. 2016A Versatile Scene Model with Differentiable Visibility Applied to Generative Pose Estimationcited by 42
  23. 2018Context-Aware Synthesis and Placement of Object Instancescited by 41
  24. 2017LR-GAN: Layered Recursive Generative Adversarial Networks for Image Generationcited by 36
  25. 2019Unsupervised Keypoint Learning for Guiding Class-Conditional Video Predictioncited by 36
  26. 2019MixNMatch: Multifactor Disentanglement and Encoding for Conditional Image Generationcited by 29
  27. 2018Learning Hierarchical Semantic Image Manipulation through Structured Representationscited by 27
  28. 2019Transformable Bottleneck Networkscited by 23
  29. 2020Pix2Surf: Learning Parametric 3D Surface Models of Objects from Imagescited by 22
  30. 2016Real-Time Hand Tracking Using a Sum of Anisotropic Gaussians Modelcited by 15
  31. 2017Hierarchical Implicit Models and Likelihood-Free Variational Inferencecited by 13
  32. 2020Reposing Humans by Warping 3D Featurescited by 13
  33. 2021Discovering Relationships between Object Categories via Universal Canonical Mapscited by 13
  34. 2021Flow Guided Transformable Bottleneck Networks for Motion Retargetingcited by 12
  35. 2018SEIGAN: Towards Compositional Image Generation by Simultaneously Learning to Segment, Enhance, and Inpaintcited by 9
  36. 2019A Layer-Based Sequential Framework for Scene Generation with GANscited by 7
  37. 2019Self-supervised Learning of 3D Objects from Natural Imagescited by 3

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 2.

5 papers.

  1. 2022EpiGRAF: Rethinking training of 3D GANscited by 108
  2. 2023GauFRe: Gaussian Deformation Fields for Real-time Dynamic Novel View Synthesiscited by 95
  3. 2023Gaussian Shell Maps for Efficient 3D Human Generationcited by 61
  4. 20223D GAN Inversion with Facial Symmetry Priorcited by 39
  5. 2022BlobGAN: Spatially Disentangled Scene Representationscited by 35