Consistent Multimodal Generation via A Unified GAN Framework
Not on the map. Why?
Abstract (arXiv)
We investigate how to generate multimodal image outputs, such as RGB, depth, and surface normals, with a single generative model. The challenge is to produce outputs that are realistic, and also consistent with each other. Our solution builds on the StyleGAN3 architecture, with a shared backbone and modality-specific branches in the last layers of the synthesis network, and we propose per-modality fidelity discriminators and a cross-modality consistency discriminator. In experiments on the Stanford2D3D dataset, we demonstrate realistic and consistent generation of RGB, depth, and normal images. We also show a training recipe to easily extend our pretrained model on a new domain, even with a few pairwise data. We further evaluate the use of synthetically generated RGB and depth pairs for training or fine-tuning depth estimators. Code will be available at https://github.com/jessemelpolio/MultimodalGAN.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 34 of its 39 references that are in the map. First posted on arXiv 4 Jul 2023.
Builds on
34 of 39 references.
References · 34
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
- 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
- 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
- 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
- 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
- 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,988
- 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
- 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,355
- 2014Depth Map Prediction from a Single Image using a Multi-Scale Deep Networkcited by 1,274
- 2020Training Generative Adversarial Networks with Limited Datacited by 1,186
- 2021Alias-Free Generative Adversarial Networkscited by 1,184
22 more
- 2017A Survey on Multi-Task Learningcited by 840
- 2017Unsupervised Learning of Depth and Ego-Motion from Videocited by 782
- 2018Which Training Methods for GANs do actually Converge?cited by 679
- 2018Taskonomy: Disentangling Task Transfer Learningcited by 634
- 2017Joint 2D-3D-Semantic Data for Indoor Scene Understandingcited by 406
- 2020Multi-Task Learning with Deep Neural Networks: A Surveycited by 375
- 2020Differentiable Augmentation for Data-Efficient GAN Trainingcited by 363
- 2021Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scanscited by 336
- 2021DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effortcited by 264
- 2020Learning to Recover 3D Scene Shape from a Single Imagecited by 216
- 2018Depth Prediction Without the Sensors: Leveraging Structure for Unsupervised Learning from Monocular Videoscited by 155
- 2021Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalizationcited by 141
- 20223D Common Corruptions and Data Augmentationcited by 119
- 2020Robust Learning Through Cross-Task Consistencycited by 105
- 2014Designing Deep Networks for Surface Normal Estimationcited by 59
- 2016Generative Image Modeling using Style and Structure Adversarial Networkscited by 50
- 2016SE3-Nets: Learning Rigid Body Motion using Deep Neural Networkscited by 49
- 20223D-Aware Indoor Scene Synthesis with Depth Priorscited by 32
- 2020Taskology: Utilizing Task Relations at Scalecited by 19
- 2021Generative Modeling for Multi-task Visual Learningcited by 14
- 2019RGBD-GAN: Unsupervised 3D Representation Learning From Natural Image Datasets via RGBD Image Synthesiscited by 13
- 2022Polymorphic-GAN: Generating Aligned Samples across Multiple Domains with Learned Morph Mapscited by 3
Built on by
Citing papers per year: 2024: 1; 2025: 2.
3 papers.