# Transformation-Grounded Image Generation Network for Novel 3D View Synthesis

arXiv: https://arxiv.org/abs/1703.02921 · HTML: https://mlsanity.com/paper/1703.02921/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Mar 2017.

## Built on by (most cited first, 30 of 59)

- [Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior](https://mlsanity.com/paper/2303.14184/) (2023)
- [Novel View Synthesis with Diffusion Models](https://mlsanity.com/paper/2210.04628/) (2022)
- [Consistent View Synthesis with Pose-Guided Diffusion Models](https://mlsanity.com/paper/2303.17598/) (2023)
- [Geometry-Free View Synthesis: Transformers and no 3D Priors](https://mlsanity.com/paper/2104.07652/) (2021)
- [FoV-NeRF: Foveated Neural Radiance Fields for Virtual Reality](https://mlsanity.com/paper/2103.16365/) (2021)
- [NVS-Solver: Video Diffusion Model as Zero-Shot Novel View Synthesizer](https://mlsanity.com/paper/2405.15364/) (2024)
- [GeoSim: Realistic Video Simulation via Geometry-Aware Composition for Self-Driving](https://mlsanity.com/paper/2101.06543/) (2021)
- [Free3D: Consistent Novel View Synthesis without 3D Representation](https://mlsanity.com/paper/2312.04551/) (2023)
- [GAN-Supervised Dense Visual Alignment](https://mlsanity.com/paper/2112.05143/) (2021)
- [Control-NeRF: Editable Feature Volumes for Scene Rendering and Manipulation](https://mlsanity.com/paper/2204.10850/) (2022)
- [Amodal3R: Amodal 3D Reconstruction from Occluded 2D Images](https://mlsanity.com/paper/2503.13439/) (2025)
- [ARAPReg: An As-Rigid-As Possible Regularization Loss for Learning Deformable Shape Generators](https://mlsanity.com/paper/2108.09432/) (2021)
- [CustomNet: Zero-shot Object Customization with Variable-Viewpoints in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2310.19784/) (2023)
- [DSO: Aligning 3D Generators with Simulation Feedback for Physical Soundness](https://mlsanity.com/paper/2503.22677/) (2025)
- [Cascaded Cross MLP-Mixer GANs for Cross-View Image Translation](https://mlsanity.com/paper/2110.10183/) (2021)
- [Cross-View Exocentric to Egocentric Video Synthesis](https://mlsanity.com/paper/2107.03120/) (2021)
- [3D-Aware Video Generation](https://mlsanity.com/paper/2206.14797/) (2022)
- [Decoupling Dynamic Monocular Videos for Dynamic View Synthesis](https://mlsanity.com/paper/2304.01716/) (2023)
- [Layout-Guided Novel View Synthesis from a Single Indoor Panorama](https://mlsanity.com/paper/2103.17022/) (2021)
- [Comparing Correspondences: Video Prediction with Correspondence-wise Losses](https://mlsanity.com/paper/2104.09498/) (2021)
- [Detailed Avatar Recovery from Single Image](https://mlsanity.com/paper/2108.02931/) (2021)
- [Novel View Synthesis with Pixel-Space Diffusion Models](https://mlsanity.com/paper/2411.07765/) (2024)
- [Self-Supervised Visibility Learning for Novel View Synthesis](https://mlsanity.com/paper/2103.15407/) (2021)
- [CBARF: Cascaded Bundle-Adjusting Neural Radiance Fields from Imperfect Camera Poses](https://mlsanity.com/paper/2310.09776/) (2023)
- [Deep View Synthesis via Self-Consistent Generative Network](https://mlsanity.com/paper/2101.10844/) (2021)
- [Learning Generalizable Light Field Networks from Few Images](https://mlsanity.com/paper/2207.11757/) (2022)
- [Learning Spatial and Spatio-Temporal Pixel Aggregations for Image and Video Denoising](https://mlsanity.com/paper/2101.10760/) (2021)
- [Unsupervised Novel View Synthesis from a Single Image](https://mlsanity.com/paper/2102.03285/) (2021)
- [Controllable Radiance Fields for Dynamic Face Synthesis](https://mlsanity.com/paper/2210.05825/) (2022)
- [BlobGAN-3D: A Spatially-Disentangled 3D-Aware Generative Model for Indoor Scenes](https://mlsanity.com/paper/2303.14706/) (2023)

## Abstract (arXiv)

We present a transformation-grounded image generation network for novel 3D view synthesis from a single image. Instead of taking a 'blank slate' approach, we first explicitly infer the parts of the geometry visible both in the input and novel views and then re-cast the remaining synthesis problem as image completion. Specifically, we both predict a flow to move the pixels from the input to the novel view along with a novel visibility map that helps deal with occulsion/disocculsion. Next, conditioned on those intermediate results, we hallucinate (infer) parts of the object invisible in the input image. In addition to the new network structure, training with a combination of adversarial and perceptual loss results in a reduction in common artifacts of novel view synthesis such as distortions and holes, while successfully generating high frequency details and preserving visual aspects of the input image. We evaluate our approach on a wide range of synthetic and real examples. Both qualitative and quantitative results show our method achieves significantly better results compared to existing methods.