# RFBNet: Deep Multimodal Networks with Residual Fusion Blocks for RGB-D Semantic Segmentation

arXiv: https://arxiv.org/abs/1907.00135 · HTML: https://mlsanity.com/paper/1907.00135/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jun 2019.

## Built on by (most cited first, 12 of 12)

- [Bidirectional Projection Network for Cross Dimension Scene Understanding](https://mlsanity.com/paper/2103.14326/) (2021)
- [3D-to-2D Distillation for Indoor Scene Parsing](https://mlsanity.com/paper/2104.02243/) (2021)
- [Missing Modality Robustness in Semi-Supervised Multi-Modal Semantic Segmentation](https://mlsanity.com/paper/2304.10756/) (2023)
- [ChatGPT-Like Large-Scale Foundation Models for Prognostics and Health Management: A Survey and Roadmaps](https://mlsanity.com/paper/2305.06472/) (2023)
- [Modality-Incremental Learning with Disjoint Relevance Mapping Networks for Image-based Semantic Segmentation](https://mlsanity.com/paper/2411.17610/) (2024)
- [Deep feature selection-and-fusion for RGB-D semantic segmentation](https://mlsanity.com/paper/2105.04102/) (2021)
- [MTMMC: A Large-Scale Real-World Multi-Modal Camera Tracking Benchmark](https://mlsanity.com/paper/2403.20225/) (2024)
- [Lightweight integration of 3D features to improve 2D image segmentation](https://mlsanity.com/paper/2212.08334/) (2022)
- [Evaluation of Multimodal Semantic Segmentation using RGB-D Data](https://mlsanity.com/paper/2103.16758/) (2021)
- [Towards Semi-supervised Dual-modal Semantic Segmentation](https://mlsanity.com/paper/2409.13325/) (2024)
- [Multi-modal 3D Pose and Shape Estimation with Computed Tomography](https://mlsanity.com/paper/2503.19405/) (2025)
- [MSNN-LINet: Cross-Modal Learning via Continuous Linear Integration](https://mlsanity.com/paper/2606.31135/) (2026)

## Abstract (arXiv)

RGB-D semantic segmentation methods conventionally use two independent encoders to extract features from the RGB and depth data. However, there lacks an effective fusion mechanism to bridge the encoders, for the purpose of fully exploiting the complementary information from multiple modalities. This paper proposes a novel bottom-up interactive fusion structure to model the interdependencies between the encoders. The structure introduces an interaction stream to interconnect the encoders. The interaction stream not only progressively aggregates modality-specific features from the encoders but also computes complementary features for them. To instantiate this structure, the paper proposes a residual fusion block (RFB) to formulate the interdependences of the encoders. The RFB consists of two residual units and one fusion unit with gate mechanism. It learns complementary features for the modality-specific encoders and extracts modality-specific features as well as cross-modal features. Based on the RFB, the paper presents the deep multimodal networks for RGB-D semantic segmentation called RFBNet. The experiments on two datasets demonstrate the effectiveness of modeling the interdependencies and that the RFBNet achieved state-of-the-art performance.