# RGBD-Net: Predicting color and depth images for novel views synthesis

arXiv: https://arxiv.org/abs/2011.14398 · HTML: https://mlsanity.com/paper/2011.14398/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Nov 2020.

## Built on by (most cited first, 11 of 11)

- [GeoNeRF: Generalizing NeRF with Geometry Priors](https://mlsanity.com/paper/2111.13539/) (2021)
- [Lightplane: Highly-Scalable Components for Neural 3D Fields](https://mlsanity.com/paper/2404.19760/) (2024)
- [Free-Viewpoint RGB-D Human Performance Capture and Rendering](https://mlsanity.com/paper/2112.13889/) (2021)
- [3inGAN: Learning a 3D Generative Model from Images of a Self-similar Scene](https://mlsanity.com/paper/2211.14902/) (2022)
- [Cascaded and Generalizable Neural Radiance Fields for Fast View Synthesis](https://mlsanity.com/paper/2208.04717/) (2022)
- [CombiNeRF: A Combination of Regularization Techniques for Few-Shot Neural Radiance Field View Synthesis](https://mlsanity.com/paper/2403.14412/) (2024)
- [GOEmbed: Gradient Origin Embeddings for Representation Agnostic 3D Feature Learning](https://mlsanity.com/paper/2312.08744/) (2023)
- [TimeNeRF: Building Generalizable Neural Radiance Fields across Time from Few-Shot Input Views](https://mlsanity.com/paper/2507.13929/) (2025)
- [FaDIV-Syn: Fast Depth-Independent View Synthesis using Soft Masks and Implicit Blending](https://mlsanity.com/paper/2106.13139/) (2021)
- [Learning Pose Image Manifolds Using Geometry-Preserving GANs and Elasticae](https://mlsanity.com/paper/2305.10513/) (2023)
- [A Novel Downsampling Strategy Based on Information Complementarity for Medical Image Segmentation](https://mlsanity.com/paper/2507.14790/) (2025)

## Abstract (arXiv)

We propose a new cascaded architecture for novel view synthesis, called RGBD-Net, which consists of two core components: a hierarchical depth regression network and a depth-aware generator network. The former one predicts depth maps of the target views by using adaptive depth scaling, while the latter one leverages the predicted depths and renders spatially and temporally consistent target images. In the experimental evaluation on standard datasets, RGBD-Net not only outperforms the state-of-the-art by a clear margin, but it also generalizes well to new scenes without per-scene optimization. Moreover, we show that RGBD-Net can be optionally trained without depth supervision while still retaining high-quality rendering. Thanks to the depth regression network, RGBD-Net can be also used for creating dense 3D point clouds that are more accurate than those produced by some state-of-the-art multi-view stereo methods.