# Learning Common and Specific Features for RGB-D Semantic Segmentation with Deconvolutional Networks

arXiv: https://arxiv.org/abs/1608.01082 · HTML: https://mlsanity.com/paper/1608.01082/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Aug 2016.

## Built on by (most cited first, 24 of 24)

- [Provable Dynamic Fusion for Low-Quality Multimodal Data](https://mlsanity.com/paper/2306.02050/) (2023)
- [The Modality Focusing Hypothesis: Towards Understanding Crossmodal Knowledge Distillation](https://mlsanity.com/paper/2206.06487/) (2022)
- [RGB-T Semantic Segmentation with Location, Activation, and Sharpening](https://mlsanity.com/paper/2210.14530/) (2022)
- [Audio-Visual Segmentation by Exploring Cross-Modal Mutual Semantics](https://mlsanity.com/paper/2307.16620/) (2023)
- [Learning Modality-agnostic Representation for Semantic Segmentation from Any Modalities](https://mlsanity.com/paper/2407.11351/) (2024)
- [Deep Learning Based 3D Segmentation: A Survey](https://mlsanity.com/paper/2103.05423/) (2021)
- [Learning Deep Multimodal Feature Representation with Asymmetric Multi-layer Fusion](https://mlsanity.com/paper/2108.05009/) (2021)
- [Efficient Multimodal Semantic Segmentation via Dual-Prompt Learning](https://mlsanity.com/paper/2312.00360/) (2023)
- [Centering the Value of Every Modality: Towards Efficient and Resilient Modality-agnostic Semantic Segmentation](https://mlsanity.com/paper/2407.11344/) (2024)
- [UniS-MMC: Multimodal Classification via Unimodality-supervised Multimodal Contrastive Learning](https://mlsanity.com/paper/2305.09299/) (2023)
- [Channel Exchanging Networks for Multimodal and Multitask Dense Image Prediction](https://mlsanity.com/paper/2112.02252/) (2021)
- [Depth-Adapted CNNs for RGB-D Semantic Segmentation](https://mlsanity.com/paper/2206.03939/) (2022)
- [ALSTER: A Local Spatio-Temporal Expert for Online 3D Semantic Reconstruction](https://mlsanity.com/paper/2311.18068/) (2023)
- [Part-Whole Relational Fusion Towards Multi-Modal Scene Understanding](https://mlsanity.com/paper/2410.14944/) (2024)
- [Deep feature selection-and-fusion for RGB-D semantic segmentation](https://mlsanity.com/paper/2105.04102/) (2021)
- [What Makes for Robust Multi-Modal Models in the Face of Missing Modalities?](https://mlsanity.com/paper/2310.06383/) (2023)
- [Towards the Causal Complete Cause of Multi-Modal Representation Learning](https://mlsanity.com/paper/2407.14058/) (2024)
- [Volumetric Mapping with Panoptic Refinement via Kernel Density Estimation for Mobile Robots](https://mlsanity.com/paper/2412.11241/) (2024)
- [Automatic Detection of Alzheimer's Disease with Multi-Modal Fusion of Clinical MRI Scans](https://mlsanity.com/paper/2311.18245/) (2023)
- [Deep Learning-Based 3D Instance and Semantic Segmentation: A Review](https://mlsanity.com/paper/2406.13308/) (2024)
- [CSFNet: A Cosine Similarity Fusion Network for Real-Time RGB-X Semantic Segmentation of Driving Scenes](https://mlsanity.com/paper/2407.01328/) (2024)
- [MaskAdapt: Unsupervised Geometry-Aware Domain Adaptation Using Multimodal Contextual Learning and RGB-Depth Masking](https://mlsanity.com/paper/2505.24026/) (2025)
- [Countering Multi-modal Representation Collapse through Rank-targeted Fusion](https://mlsanity.com/paper/2511.06450/) (2025)
- [Fully Exploiting Vision Foundation Model's Profound Prior Knowledge for Generalizable RGB-Depth Driving Scene Parsing](https://mlsanity.com/paper/2502.06219/) (2025)

## Abstract (arXiv)

In this paper, we tackle the problem of RGB-D semantic segmentation of indoor images. We take advantage of deconvolutional networks which can predict pixel-wise class labels, and develop a new structure for deconvolution of multiple modalities. We propose a novel feature transformation network to bridge the convolutional networks and deconvolutional networks. In the feature transformation network, we correlate the two modalities by discovering common features between them, as well as characterize each modality by discovering modality specific features. With the common features, we not only closely correlate the two modalities, but also allow them to borrow features from each other to enhance the representation of shared information. With specific features, we capture the visual patterns that are only visible in one modality. The proposed network achieves competitive segmentation accuracy on NYU depth dataset V1 and V2.