# AUTO3D: Novel view synthesis through unsupervisely learned variational viewpoint and global 3D representation

arXiv: https://arxiv.org/abs/2007.06620 · HTML: https://mlsanity.com/paper/2007.06620/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Jul 2020.

## Built on by (most cited first, 14 of 14)

- [Subtype-aware Unsupervised Domain Adaptation for Medical Diagnosis](https://mlsanity.com/paper/2101.00318/) (2021)
- [Energy-constrained Self-training for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2101.00316/) (2021)
- [Dual-cycle Constrained Bijective VAE-GAN For Tagged-to-Cine Magnetic Resonance Image Synthesis](https://mlsanity.com/paper/2101.05439/) (2021)
- [Vision-based Neural Scene Representations for Spacecraft](https://mlsanity.com/paper/2105.06405/) (2021)
- [Identity-aware Facial Expression Recognition in Compressed Video](https://mlsanity.com/paper/2101.00317/) (2021)
- [Unsupervised Novel View Synthesis from a Single Image](https://mlsanity.com/paper/2102.03285/) (2021)
- [Variational Inference for Quantifying Inter-observer Variability in Segmentation of Anatomical Structures](https://mlsanity.com/paper/2201.07106/) (2022)
- [WildRayZer: Self-supervised Large View Synthesis in Dynamic Environments](https://mlsanity.com/paper/2601.10716/) (2026)
- [Structure-aware Unsupervised Tagged-to-Cine MRI Synthesis with Self Disentanglement](https://mlsanity.com/paper/2202.12474/) (2022)
- [Subtype-Aware Dynamic Unsupervised Domain Adaptation](https://mlsanity.com/paper/2208.07754/) (2022)
- [Novel View Synthesis from a Single Image via Unsupervised learning](https://mlsanity.com/paper/2110.15569/) (2021)
- [Tagged-MRI Sequence to Audio Synthesis via Self Residual Attention Guided Heterogeneous Translator](https://mlsanity.com/paper/2206.02284/) (2022)
- [Single-view Neural Radiance Fields with Depth Teacher](https://mlsanity.com/paper/2303.09952/) (2023)
- [Treatment-wise Glioblastoma Survival Inference with Multi-parametric Preoperative MRI](https://mlsanity.com/paper/2402.06982/) (2024)

## Abstract (arXiv)

This paper targets on learning-based novel view synthesis from a single or limited 2D images without the pose supervision. In the viewer-centered coordinates, we construct an end-to-end trainable conditional variational framework to disentangle the unsupervisely learned relative-pose/rotation and implicit global 3D representation (shape, texture and the origin of viewer-centered coordinates, etc.). The global appearance of the 3D object is given by several appearance-describing images taken from any number of viewpoints. Our spatial correlation module extracts a global 3D representation from the appearance-describing images in a permutation invariant manner. Our system can achieve implicitly 3D understanding without explicitly 3D reconstruction. With an unsupervisely learned viewer-centered relative-pose/rotation code, the decoder can hallucinate the novel view continuously by sampling the relative-pose in a prior distribution. In various applications, we demonstrate that our model can achieve comparable or even better results than pose/3D model-supervised learning-based novel view synthesis (NVS) methods with any number of input views.