# Multi-Modal RGB-D Scene Recognition Across Domains

arXiv: https://arxiv.org/abs/2103.14672 · HTML: https://mlsanity.com/paper/2103.14672/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 40 references that are in the map.
- First posted on arXiv 26 Mar 2021.

## Builds on (most cited first, 20 of 20)

- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Learning Transferable Features with Deep Adaptation Networks](https://mlsanity.com/paper/1502.02791/) (2015)
- [Adversarial Discriminative Domain Adaptation](https://mlsanity.com/paper/1702.05464/) (2017)
- [Striving for Simplicity: The All Convolutional Net](https://mlsanity.com/paper/1412.6806/) (2014)
- [Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks](https://mlsanity.com/paper/1703.10593/) (2017)
- [Learning to Adapt Structured Output Space for Semantic Segmentation](https://mlsanity.com/paper/1802.10349/) (2018)
- [Return of Frustratingly Easy Domain Adaptation](https://mlsanity.com/paper/1511.05547/) (2015)
- [Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks](https://mlsanity.com/paper/1612.05424/) (2016)
- [Cross Modal Distillation for Supervision Transfer](https://mlsanity.com/paper/1507.00448/) (2015)
- [Learning Rich Features from RGB-D Images for Object Detection and Segmentation](https://mlsanity.com/paper/1407.5736/) (2014)
- [Self-supervised Domain Adaptation for Computer Vision Tasks](https://mlsanity.com/paper/1907.10915/) (2019)
- [Indoor Semantic Segmentation using depth information](https://mlsanity.com/paper/1301.3572/) (2013)
- [From source to target and back: symmetric bi-directional adaptive GAN](https://mlsanity.com/paper/1705.08824/) (2017)
- [Self-Supervised Learning Across Domains](https://mlsanity.com/paper/2007.12368/) (2020)
- [Translate-to-Recognize Networks for RGB-D Scene Recognition](https://mlsanity.com/paper/1904.12254/) (2019)
- [Unsupervised Domain Adaptation through Inter-modal Rotation for RGB-D Object Recognition](https://mlsanity.com/paper/2004.10016/) (2020)
- [YCB-M: A Multi-Camera RGB-D Dataset for Object Recognition and 6DoF Pose Estimation](https://mlsanity.com/paper/2004.11657/) (2020)
- [Learning Effective RGB-D Representations for Scene Recognition](https://mlsanity.com/paper/1809.06269/) (2018)
- [Deep Learning for Scene Classification: A Survey](https://mlsanity.com/paper/2101.10531/) (2021)
- [Unsupervised Domain Adaptation Learning Algorithm for RGB-D Staircase Recognition](https://mlsanity.com/paper/1903.01212/) (2019)

## Built on by (most cited first, 5 of 5)

- [Provable Dynamic Fusion for Low-Quality Multimodal Data](https://mlsanity.com/paper/2306.02050/) (2023)
- [Multimodal Across Domains Gaze Target Detection](https://mlsanity.com/paper/2208.10822/) (2022)
- [DisCoM-KD: Cross-Modal Knowledge Distillation via Disentanglement Representation and Adversarial Learning](https://mlsanity.com/paper/2408.07080/) (2024)
- [Source-Free Cross-Modal Knowledge Transfer by Unleashing the Potential of Task-Irrelevant Data](https://mlsanity.com/paper/2401.05014/) (2024)
- [LUWA Dataset: Learning Lithic Use-Wear Analysis on Microscopic Images](https://mlsanity.com/paper/2403.13171/) (2024)

## Abstract (arXiv)

Scene recognition is one of the basic problems in computer vision research with extensive applications in robotics. When available, depth images provide helpful geometric cues that complement the RGB texture information and help to identify discriminative scene image features. Depth sensing technology developed fast in the last years and a great variety of 3D cameras have been introduced, each with different acquisition properties. However, those properties are often neglected when targeting big data collections, so multi-modal images are gathered disregarding their original nature. In this work, we put under the spotlight the existence of a possibly severe domain shift issue within multi-modality scene recognition datasets. As a consequence, a scene classification model trained on one camera may not generalize on data from a different camera, only providing a low recognition performance. Starting from the well-known SUN RGB-D dataset, we designed an experimental testbed to study this problem and we use it to benchmark the performance of existing methods. Finally, we introduce a novel adaptive scene recognition approach that leverages self-supervised translation between modalities. Indeed, learning to go from RGB to depth and vice-versa is an unsupervised procedure that can be trained jointly on data of multiple cameras and may help to bridge the gap among the extracted feature distributions. Our experimental results confirm the effectiveness of the proposed approach.