# RGBD20K: A Large-Scale Benchmark for RGB-D Semantic Segmentation

arXiv: https://arxiv.org/abs/2609.29028 · HTML: https://mlsanity.com/paper/2609.29028/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 40 references that are in the map.
- First posted on arXiv 24 Sep 2026.

## Builds on (most cited first, 25 of 25)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)
- [Joint 2D-3D-Semantic Data for Indoor Scene Understanding](https://mlsanity.com/paper/1702.01105/) (2017)
- [MultiMAE: Multi-modal Multi-task Masked Autoencoders](https://mlsanity.com/paper/2204.01678/) (2022)
- [CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers](https://mlsanity.com/paper/2203.04838/) (2022)
- [Multimodal Token Fusion for Vision Transformers](https://mlsanity.com/paper/2204.08721/) (2022)
- [Delivering Arbitrary-Modal Semantic Segmentation](https://mlsanity.com/paper/2303.01480/) (2023)
- [PST900: RGB-Thermal Calibration, Dataset and Segmentation Network](https://mlsanity.com/paper/1909.10980/) (2019)
- [DepthTrack : Unveiling the Power of RGBD Tracking](https://mlsanity.com/paper/2108.13962/) (2021)
- [DFormer: Rethinking RGBD Representation Learning for Semantic Segmentation](https://mlsanity.com/paper/2309.09668/) (2023)
- [DIML/CVL RGB-D Dataset: 2M RGB-D Images of Natural Indoor and Outdoor Scenes](https://mlsanity.com/paper/2110.11590/) (2021)
- [Edge-aware Guidance Fusion Network for RGB Thermal Scene Parsing](https://mlsanity.com/paper/2112.05144/) (2021)
- [GeminiFusion: Efficient Pixel-wise Multimodal Fusion for Vision Transformer](https://mlsanity.com/paper/2406.01210/) (2024)
- [RGBD1K: A Large-scale Dataset and Benchmark for RGB-D Object Tracking](https://mlsanity.com/paper/2208.09787/) (2022)
- [DFormerv2: Geometry Self-Attention for RGBD Semantic Segmentation](https://mlsanity.com/paper/2504.04701/) (2025)
- [VastTrack: Vast Category Visual Object Tracking](https://mlsanity.com/paper/2403.03493/) (2024)
- [Efficient Multimodal Semantic Segmentation via Dual-Prompt Learning](https://mlsanity.com/paper/2312.00360/) (2023)
- [ARKitTrack: A New Diverse Dataset for Tracking Using Mobile RGB-D Data](https://mlsanity.com/paper/2303.13885/) (2023)
- [ViDSOD-100: A New Dataset and a Baseline Model for RGB-D Video Salient Object Detection](https://mlsanity.com/paper/2406.12536/) (2024)
- [LoReTrack: Efficient and Accurate Low-Resolution Transformer Tracking](https://mlsanity.com/paper/2405.17660/) (2024)
- [DMTrack: Spatio-Temporal Multimodal Tracking via Dual-Adapter](https://mlsanity.com/paper/2508.01592/) (2025)

## Abstract (arXiv)

In this paper, we propose RGBD20K, a novel dataset for facilitating the development of more robust and general RGB-D semantic segmentation by encompassing abundant categories and high-quality annotations. RGBD20K possesses several attractive properties: (1) Expanded Semantic Space. In particular, it covers 160 fine-grained categories, largely surpassing the category diversity of existing popular RGB-D benchmarks (e.g., NYUv2 with 40 classes and SUN RGB-D with 37 classes). With such enriched semantic coverage, we expect to promote the learning of more generalizable segmentation models. (2) Larger Scale. Compared with current benchmarks, RGBD20K offers 20,000 RGB-D image pairs, providing a substantially larger training resource that benefits the development of more powerful deep models. (3) High-Fidelity Annotation. We perform rigorous re-evaluation and correction of existing labels to resolve long-standing annotation noise, resulting in a clean and reliable ground-truth foundation. Furthermore, we propose a novel score-purified fusion (SPF) method, which achieves state-of-the-art performance across all evaluated benchmarks, demonstrating the effectiveness of our approach in leveraging high-quality multimodal information for RGB-D semantic segmentation. The dataset is here: https://github.com/ShaohuaDong2021/RGBD20K/.