# Multi-Modal Attention-based Fusion Model for Semantic Segmentation of RGB-Depth Images

arXiv: https://arxiv.org/abs/1912.11691 · HTML: https://mlsanity.com/paper/1912.11691/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Dec 2019.

## Built on by (most cited first, 13 of 13)

- [CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers](https://mlsanity.com/paper/2203.04838/) (2022)
- [ShapeConv: Shape-aware Convolutional Layer for Indoor RGB-D Semantic Segmentation](https://mlsanity.com/paper/2108.10528/) (2021)
- [Expansion-Squeeze-Excitation Fusion Network for Elderly Activity Recognition](https://mlsanity.com/paper/2112.10992/) (2021)
- [SemAttNet: Towards Attention-based Semantic Aware Guided Depth Completion](https://mlsanity.com/paper/2204.13635/) (2022)
- [Pixel Difference Convolutional Network for RGB-D Semantic Segmentation](https://mlsanity.com/paper/2302.11951/) (2023)
- [DCANet: Differential Convolution Attention Network for RGB-D Semantic Segmentation](https://mlsanity.com/paper/2210.06747/) (2022)
- [Efficient Multi-Task Scene Analysis with RGB-D Transformers](https://mlsanity.com/paper/2306.05242/) (2023)
- [ComPtr: Towards Diverse Bi-source Dense Prediction Tasks via A Simple yet General Complementary Transformer](https://mlsanity.com/paper/2307.12349/) (2023)
- [ByDeWay: Boost Your multimodal LLM with DEpth prompting in a Training-Free Way](https://mlsanity.com/paper/2507.08679/) (2025)
- [Depth-guided Texture Diffusion for Image Semantic Segmentation](https://mlsanity.com/paper/2408.09097/) (2024)
- [Optical Flow Matters: an Empirical Comparative Study on Fusing Monocular Extracted Modalities for Better Steering](https://mlsanity.com/paper/2409.12716/) (2024)
- [COP-GEN-Beta: Unified Generative Modelling of COPernicus Imagery Thumbnails](https://mlsanity.com/paper/2504.08548/) (2025)
- [SPDA-SAM: A Self-prompted Depth-Aware Segment Anything Model for Instance Segmentation](https://mlsanity.com/paper/2602.06335/) (2026)

## Abstract (arXiv)

The 3D scene understanding is mainly considered as a crucial requirement in computer vision and robotics applications. One of the high-level tasks in 3D scene understanding is semantic segmentation of RGB-Depth images. With the availability of RGB-D cameras, it is desired to improve the accuracy of the scene understanding process by exploiting the depth features along with the appearance features. As depth images are independent of illumination, they can improve the quality of semantic labeling alongside RGB images. Consideration of both common and specific features of these two modalities improves the performance of semantic segmentation. One of the main problems in RGB-Depth semantic segmentation is how to fuse or combine these two modalities to achieve more advantages of each modality while being computationally efficient. Recently, the methods that encounter deep convolutional neural networks have reached the state-of-the-art results by early, late, and middle fusion strategies. In this paper, an efficient encoder-decoder model with the attention-based fusion block is proposed to integrate mutual influences between feature maps of these two modalities. This block explicitly extracts the interdependences among concatenated feature maps of these modalities to exploit more powerful feature maps from RGB-Depth images. The extensive experimental results on three main challenging datasets of NYU-V2, SUN RGB-D, and Stanford 2D-3D-Semantic show that the proposed network outperforms the state-of-the-art models with respect to computational cost as well as model size. Experimental results also illustrate the effectiveness of the proposed lightweight attention-based fusion model in terms of accuracy.