# A Survey on RGB-D Datasets

arXiv: https://arxiv.org/abs/2201.05761 · HTML: https://mlsanity.com/paper/2201.05761/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 101 of its 265 references that are in the map.
- First posted on arXiv 15 Jan 2022.

## Builds on (most cited first, 30 of 101)

- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [The Replica Dataset: A Digital Replica of Indoor Spaces](https://mlsanity.com/paper/1906.05797/) (2019)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [MegaDepth: Learning Single-View Depth Prediction from Internet Photos](https://mlsanity.com/paper/1804.00607/) (2018)
- [Argoverse: 3D Tracking and Forecasting with Rich Maps](https://mlsanity.com/paper/1911.02620/) (2019)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding](https://mlsanity.com/paper/2011.02523/) (2020)
- [Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI](https://mlsanity.com/paper/2109.08238/) (2021)
- [KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D](https://mlsanity.com/paper/2109.13410/) (2021)
- [NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding](https://mlsanity.com/paper/1905.04757/) (2019)
- [Habitat 2.0: Training Home Assistants to Rearrange their Habitat](https://mlsanity.com/paper/2106.14405/) (2021)
- [TartanAir: A Dataset to Push the Limits of Visual SLAM](https://mlsanity.com/paper/2003.14338/) (2020)
- [BlendedMVS: A Large-scale Dataset for Generalized Multi-view Stereo Networks](https://mlsanity.com/paper/1911.10127/) (2019)
- [Gibson Env: Real-World Perception for Embodied Agents](https://mlsanity.com/paper/1808.10654/) (2018)
- [Semantic Scene Completion from a Single Depth Image](https://mlsanity.com/paper/1611.08974/) (2016)
- [3D Packing for Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/1905.02693/) (2019)
- [Joint 2D-3D-Semantic Data for Indoor Scene Understanding](https://mlsanity.com/paper/1702.01105/) (2017)
- [Learning joint reconstruction of hands and manipulated objects](https://mlsanity.com/paper/1904.05767/) (2019)
- [Virtual KITTI 2](https://mlsanity.com/paper/2001.10773/) (2020)
- [Virtual Worlds as Proxy for Multi-Object Tracking Analysis](https://mlsanity.com/paper/1605.06457/) (2016)
- [Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scans](https://mlsanity.com/paper/2110.04994/) (2021)
- [Sparsity Invariant CNNs](https://mlsanity.com/paper/1708.06500/) (2017)

## Built on by (most cited first, 11 of 11)

- [Salient Object Detection in RGB-D Videos](https://mlsanity.com/paper/2310.15482/) (2023)
- [X-Prompt: Multi-modal Visual Prompt for Video Object Segmentation](https://mlsanity.com/paper/2409.19342/) (2024)
- [Outdoor Monocular Depth Estimation: A Research Review](https://mlsanity.com/paper/2205.01399/) (2022)
- [D4D: An RGBD diffusion model to boost monocular depth estimation](https://mlsanity.com/paper/2403.07516/) (2024)
- [Enhancing Monocular Depth Estimation with Multi-Source Auxiliary Tasks](https://mlsanity.com/paper/2501.12824/) (2025)
- [ColabSfM: Collaborative Structure-from-Motion by Point Cloud Registration](https://mlsanity.com/paper/2503.17093/) (2025)
- [AutoDepthNet: High Frame Rate Depth Map Reconstruction using Commodity Depth and RGB Cameras](https://mlsanity.com/paper/2305.14731/) (2023)
- [Depth-aware Fusion Method based on Image and 4D Radar Spectrum for 3D Object Detection](https://mlsanity.com/paper/2502.15516/) (2025)
- [Shaken, Not Stirred: A Novel Dataset for Visual Understanding of Glasses in Human-Robot Bartending Tasks](https://mlsanity.com/paper/2503.04308/) (2025)
- [HiddenObject: Modality-Agnostic Fusion for Multimodal Hidden Object Detection](https://mlsanity.com/paper/2508.21135/) (2025)
- [Filling the Pareto-Optimal Front for Affordance Segmentation on Embedded Devices Using RGB-D Cameras](https://mlsanity.com/paper/2607.28293/) (2026)

## Abstract (arXiv)

RGB-D data is essential for solving many problems in computer vision. Hundreds of public RGB-D datasets containing various scenes, such as indoor, outdoor, aerial, driving, and medical, have been proposed. These datasets are useful for different applications and are fundamental for addressing classic computer vision tasks, such as monocular depth estimation. This paper reviewed and categorized image datasets that include depth information. We gathered 203 datasets that contain accessible data and grouped them into three categories: scene/objects, body, and medical. We also provided an overview of the different types of sensors, depth applications, and we examined trends and future directions of the usage and creation of datasets containing depth data, and how they can be applied to investigate the development of generalizable machine learning models in the monocular depth estimation field.