# Translate-to-Recognize Networks for RGB-D Scene Recognition

arXiv: https://arxiv.org/abs/1904.12254 · HTML: https://mlsanity.com/paper/1904.12254/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Apr 2019.

## Built on by (most cited first, 20 of 20)

- [Omnivore: A Single Model for Many Visual Modalities](https://mlsanity.com/paper/2201.08377/) (2022)
- [Trusted Multi-View Classification with Dynamic Evidential Fusion](https://mlsanity.com/paper/2204.11423/) (2022)
- [Semantics for Robotic Mapping, Perception and Interaction: A Survey](https://mlsanity.com/paper/2101.00443/) (2021)
- [EventDance: Unsupervised Source-free Cross-modal Adaptation for Event-based Object Recognition](https://mlsanity.com/paper/2403.14082/) (2024)
- [RGB-Depth Fusion GAN for Indoor Depth Completion](https://mlsanity.com/paper/2203.10856/) (2022)
- [Cross-Modal Knowledge Transfer Without Task-Relevant Source Data](https://mlsanity.com/paper/2209.04027/) (2022)
- [ASK: Adaptively Selecting Key Local Features for RGB-D Scene Recognition](https://mlsanity.com/paper/2110.07703/) (2021)
- [Channel Exchanging Networks for Multimodal and Multitask Dense Image Prediction](https://mlsanity.com/paper/2112.02252/) (2021)
- [Deep Learning for Scene Classification: A Survey](https://mlsanity.com/paper/2101.10531/) (2021)
- [CGiS-Net: Aggregating Colour, Geometry and Implicit Semantic Features for Indoor Place Recognition](https://mlsanity.com/paper/2202.02070/) (2022)
- [CoMAE: Single Model Hybrid Pre-training on Small-Scale RGB-D Datasets](https://mlsanity.com/paper/2302.06148/) (2023)
- [RDFC-GAN: RGB-Depth Fusion CycleGAN for Indoor Depth Completion](https://mlsanity.com/paper/2306.03584/) (2023)
- [Robots Autonomously Detecting People: A Multimodal Deep Contrastive Learning Method Robust to Intraclass Variations](https://mlsanity.com/paper/2203.00187/) (2022)
- [Multi-Modal RGB-D Scene Recognition Across Domains](https://mlsanity.com/paper/2103.14672/) (2021)
- [Source-Free Cross-Modal Knowledge Transfer by Unleashing the Potential of Task-Irrelevant Data](https://mlsanity.com/paper/2401.05014/) (2024)
- [D4D: An RGBD diffusion model to boost monocular depth estimation](https://mlsanity.com/paper/2403.07516/) (2024)
- [EventDance++: Language-guided Unsupervised Source-free Cross-modal Adaptation for Event-based Object Recognition](https://mlsanity.com/paper/2409.12778/) (2024)
- [Uncertainty Quantification via H"older Divergence for Multi-View Representation Learning](https://mlsanity.com/paper/2411.00826/) (2024)
- [Dynamic Graph Neural Network with Adaptive Features Selection for RGB-D Based Indoor Scene Recognition](https://mlsanity.com/paper/2604.00372/) (2026)
- [MSNN-LINet: Cross-Modal Learning via Continuous Linear Integration](https://mlsanity.com/paper/2606.31135/) (2026)

## Abstract (arXiv)

Cross-modal transfer is helpful to enhance modality-specific discriminative power for scene recognition. To this end, this paper presents a unified framework to integrate the tasks of cross-modal translation and modality-specific recognition, termed as Translate-to-Recognize Network (TRecgNet). Specifically, both translation and recognition tasks share the same encoder network, which allows to explicitly regularize the training of recognition task with the help of translation, and thus improve its final generalization ability. For translation task, we place a decoder module on top of the encoder network and it is optimized with a new layer-wise semantic loss, while for recognition task, we use a linear classifier based on the feature embedding from encoder and its training is guided by the standard cross-entropy loss. In addition, our TRecgNet allows to exploit large numbers of unlabeled RGB-D data to train the translation task and thus improve the representation power of encoder network. Empirically, we verify that this new semi-supervised setting is able to further enhance the performance of recognition network. We perform experiments on two RGB-D scene recognition benchmarks: NYU Depth v2 and SUN RGB-D, demonstrating that TRecgNet achieves superior performance to the existing state-of-the-art methods, especially for recognition solely based on a single modality.