# Depth CNNs for RGB-D scene recognition: learning from scratch better than transferring from RGB-CNNs

arXiv: https://arxiv.org/abs/1801.06797 · HTML: https://mlsanity.com/paper/1801.06797/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jan 2018.

## Built on by (most cited first, 7 of 7)

- [ASK: Adaptively Selecting Key Local Features for RGB-D Scene Recognition](https://mlsanity.com/paper/2110.07703/) (2021)
- [Deep Learning for Scene Classification: A Survey](https://mlsanity.com/paper/2101.10531/) (2021)
- [CoMAE: Single Model Hybrid Pre-training on Small-Scale RGB-D Datasets](https://mlsanity.com/paper/2302.06148/) (2023)
- [Attentional Graph Convolutional Network for Structure-aware Audio-Visual Scene Classification](https://mlsanity.com/paper/2301.00145/) (2022)
- [Depth as Prior Knowledge for Object Detection](https://mlsanity.com/paper/2602.05730/) (2026)
- [Dynamic Graph Neural Network with Adaptive Features Selection for RGB-D Based Indoor Scene Recognition](https://mlsanity.com/paper/2604.00372/) (2026)
- [MSNN-LINet: Cross-Modal Learning via Continuous Linear Integration](https://mlsanity.com/paper/2606.31135/) (2026)

## Abstract (arXiv)

Scene recognition with RGB images has been extensively studied and has reached very remarkable recognition levels, thanks to convolutional neural networks (CNN) and large scene datasets. In contrast, current RGB-D scene data is much more limited, so often leverages RGB large datasets, by transferring pretrained RGB CNN models and fine-tuning with the target RGB-D dataset. However, we show that this approach has the limitation of hardly reaching bottom layers, which is key to learn modality-specific features. In contrast, we focus on the bottom layers, and propose an alternative strategy to learn depth features combining local weakly supervised training from patches followed by global fine tuning with images. This strategy is capable of learning very discriminative depth-specific features with limited depth images, without resorting to Places-CNN. In addition we propose a modified CNN architecture to further match the complexity of the model and the amount of data available. For RGB-D scene recognition, depth and RGB features are combined by projecting them in a common space and further leaning a multilayer classifier, which is jointly optimized in an end-to-end network. Our framework achieves state-of-the-art accuracy on NYU2 and SUN RGB-D in both depth only and combined RGB-D data.