# SceneNet RGB-D: 5M Photorealistic Images of Synthetic Indoor Trajectories with Ground Truth

arXiv: https://arxiv.org/abs/1612.05079 · HTML: https://mlsanity.com/paper/1612.05079/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Dec 2016.

## Built on by (most cited first, 30 of 32)

- [NeRF--: Neural Radiance Fields Without Known Camera Parameters](https://mlsanity.com/paper/2102.07064/) (2021)
- [Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIP](https://mlsanity.com/paper/2208.05516/) (2022)
- [Monocular Depth Estimation using Diffusion Models](https://mlsanity.com/paper/2302.14816/) (2023)
- [SITA: Single Image Test-time Adaptation](https://mlsanity.com/paper/2112.02355/) (2021)
- [SensatUrban: Learning Semantics from Urban-Scale Photogrammetric Point Clouds](https://mlsanity.com/paper/2201.04494/) (2022)
- [ECoDepth: Effective Conditioning of Diffusion Models for Monocular Depth Estimation](https://mlsanity.com/paper/2403.18807/) (2024)
- [Learning Topology from Synthetic Data for Unsupervised Depth Completion](https://mlsanity.com/paper/2106.02994/) (2021)
- [Single Image Depth Estimation: An Overview](https://mlsanity.com/paper/2104.06456/) (2021)
- [Deep Depth Completion from Extremely Sparse Data: A Survey](https://mlsanity.com/paper/2205.05335/) (2022)
- [Scal3R: Scalable Test-Time Training for Large-Scale 3D Reconstruction](https://mlsanity.com/paper/2604.08542/) (2026)
- [Test-Time Adaptation for Depth Completion](https://mlsanity.com/paper/2402.03312/) (2024)
- [Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Models](https://mlsanity.com/paper/2501.18592/) (2025)
- [MobileBrick: Building LEGO for 3D Reconstruction on Mobile Devices](https://mlsanity.com/paper/2303.01932/) (2023)
- [Neural Radiance Field-based Visual Rendering: A Comprehensive Review](https://mlsanity.com/paper/2404.00714/) (2024)
- [Diffusion Models in 3D Vision: A Survey](https://mlsanity.com/paper/2410.04738/) (2024)
- [Vision as Unified Multimodal Generation](https://mlsanity.com/paper/2607.06560/) (2026)
- [On The State of Data In Computer Vision: Human Annotations Remain Indispensable for Developing Deep Learning Models](https://mlsanity.com/paper/2108.00114/) (2021)
- [Unsupervised Adaptation of Semantic Segmentation Models without Source Data](https://mlsanity.com/paper/2112.02359/) (2021)
- [Deep Learning on 3D Semantic Segmentation: A Detailed Review](https://mlsanity.com/paper/2411.02104/) (2024)
- [Zero-shot Depth Completion via Test-time Alignment with Affine-invariant Depth Prior](https://mlsanity.com/paper/2502.06338/) (2025)
- [Multi-task View Synthesis with Neural Radiance Fields](https://mlsanity.com/paper/2309.17450/) (2023)
- [Dense Depth Distillation with Out-of-Distribution Simulated Images](https://mlsanity.com/paper/2208.12464/) (2022)
- [ETA: Energy-based Test-time Adaptation for Depth Completion](https://mlsanity.com/paper/2508.05989/) (2025)
- [Bayesian Deep Neural Networks for Supervised Learning of Single-View Depth](https://mlsanity.com/paper/2104.14202/) (2021)
- [Review on Indoor RGB-D Semantic Segmentation with Deep Convolutional Neural Networks](https://mlsanity.com/paper/2105.11925/) (2021)
- [GCISG: Guided Causal Invariant Learning for Improved Syn-to-real Generalization](https://mlsanity.com/paper/2208.10024/) (2022)
- [Analyzing Effects of Fake Training Data on the Performance of Deep Learning Systems](https://mlsanity.com/paper/2303.01268/) (2023)
- [Uncertainty and Self-Supervision in Single-View Depth](https://mlsanity.com/paper/2406.14226/) (2024)
- [Self-supervised Monocular Depth and Pose Estimation for Endoscopy with Latent Priors](https://mlsanity.com/paper/2411.17790/) (2024)
- [Automating 3D Dataset Generation with Neural Radiance Fields](https://mlsanity.com/paper/2503.15997/) (2025)

## Abstract (arXiv)

We introduce SceneNet RGB-D, expanding the previous work of SceneNet to enable large scale photorealistic rendering of indoor scene trajectories. It provides pixel-perfect ground truth for scene understanding problems such as semantic segmentation, instance segmentation, and object detection, and also for geometric computer vision problems such as optical flow, depth estimation, camera pose estimation, and 3D reconstruction. Random sampling permits virtually unlimited scene configurations, and here we provide a set of 5M rendered RGB-D images from over 15K trajectories in synthetic layouts with random but physically simulated object poses. Each layout also has random lighting, camera trajectories, and textures. The scale of this dataset is well suited for pre-training data-driven computer vision techniques from scratch with RGB-D inputs, which previously has been limited by relatively small labelled datasets in NYUv2 and SUN RGB-D. It also provides a basis for investigating 3D scene labelling tasks by providing perfect camera poses and depth data as proxy for a SLAM system. We host the dataset at http://robotvault.bitbucket.io/scenenet-rgbd.html