# ACNet: Attention Based Network to Exploit Complementary Features for RGBD Semantic Segmentation

arXiv: https://arxiv.org/abs/1905.10089 · HTML: https://mlsanity.com/paper/1905.10089/

## Key facts

- Cited by 107 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 May 2019.

## Built on by (most cited first, 30 of 107)

- [CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers](https://mlsanity.com/paper/2203.04838/) (2022)
- [What Makes Multi-modal Learning Better than Single (Provably)](https://mlsanity.com/paper/2106.04538/) (2021)
- [Delivering Arbitrary-Modal Semantic Segmentation](https://mlsanity.com/paper/2303.01480/) (2023)
- [ShapeConv: Shape-aware Convolutional Layer for Indoor RGB-D Semantic Segmentation](https://mlsanity.com/paper/2108.10528/) (2021)
- [Sigma: Siamese Mamba Network for Multi-Modal Semantic Segmentation](https://mlsanity.com/paper/2404.04256/) (2024)
- [InverseForm: A Loss Function for Structured Boundary-Aware Segmentation](https://mlsanity.com/paper/2104.02745/) (2021)
- [DFormer: Rethinking RGBD Representation Learning for Semantic Segmentation](https://mlsanity.com/paper/2309.09668/) (2023)
- [Dynamic Multimodal Fusion](https://mlsanity.com/paper/2204.00102/) (2022)
- [On Uni-Modal Feature Learning in Supervised Multi-Modal Learning](https://mlsanity.com/paper/2305.01233/) (2023)
- [Attention, please! A survey of Neural Attention Models in Deep Learning](https://mlsanity.com/paper/2103.16775/) (2021)
- [Capturing Omni-Range Context for Omnidirectional Segmentation](https://mlsanity.com/paper/2103.05687/) (2021)
- [GeminiFusion: Efficient Pixel-wise Multimodal Fusion for Vision Transformer](https://mlsanity.com/paper/2406.01210/) (2024)
- [Improving Multi-Modal Learning with Uni-Modal Teachers](https://mlsanity.com/paper/2106.11059/) (2021)
- [Explicit Attention-Enhanced Fusion for RGB-Thermal Perception Tasks](https://mlsanity.com/paper/2303.15710/) (2023)
- [MMANet: Margin-aware Distillation and Modality-aware Regularization for Incomplete Multimodal Learning](https://mlsanity.com/paper/2304.08028/) (2023)
- [RGB-T Semantic Segmentation with Location, Activation, and Sharpening](https://mlsanity.com/paper/2210.14530/) (2022)
- [Efficient Multi-Task RGB-D Scene Analysis for Indoor Environments](https://mlsanity.com/paper/2207.04526/) (2022)
- [DFormerv2: Geometry Self-Attention for RGBD Semantic Segmentation](https://mlsanity.com/paper/2504.04701/) (2025)
- [Semantic Image Segmentation: Two Decades of Research](https://mlsanity.com/paper/2302.06378/) (2023)
- [Efficient Multimodal Semantic Segmentation via Dual-Prompt Learning](https://mlsanity.com/paper/2312.00360/) (2023)
- [Missing Modality Robustness in Semi-Supervised Multi-Modal Semantic Segmentation](https://mlsanity.com/paper/2304.10756/) (2023)
- [Cerberus Transformer: Joint Semantic, Affordance and Attribute Parsing](https://mlsanity.com/paper/2111.12608/) (2021)
- [Centering the Value of Every Modality: Towards Efficient and Resilient Modality-agnostic Semantic Segmentation](https://mlsanity.com/paper/2407.11344/) (2024)
- [Single Frame Semantic Segmentation Using Multi-Modal Spherical Images](https://mlsanity.com/paper/2308.09369/) (2023)
- [TransKD: Transformer Knowledge Distillation for Efficient Semantic Segmentation](https://mlsanity.com/paper/2202.13393/) (2022)
- [Fast Road Segmentation via Uncertainty-aware Symmetric Network](https://mlsanity.com/paper/2203.04537/) (2022)
- [AsymFormer: Asymmetrical Cross-Modal Representation Learning for Mobile Platform Real-Time RGB-D Semantic Segmentation](https://mlsanity.com/paper/2309.14065/) (2023)
- [UniRGB-IR: A Unified Framework for Visible-Infrared Semantic Tasks via Adapter Tuning](https://mlsanity.com/paper/2404.17360/) (2024)
- [Robust Multimodal Learning via Representation Decoupling](https://mlsanity.com/paper/2407.04458/) (2024)
- [Improving Multimodal Learning via Imbalanced Learning](https://mlsanity.com/paper/2507.10203/) (2025)

## Abstract (arXiv)

Compared to RGB semantic segmentation, RGBD semantic segmentation can achieve better performance by taking depth information into consideration. However, it is still problematic for contemporary segmenters to effectively exploit RGBD information since the feature distributions of RGB and depth (D) images vary significantly in different scenes. In this paper, we propose an Attention Complementary Network (ACNet) that selectively gathers features from RGB and depth branches. The main contributions lie in the Attention Complementary Module (ACM) and the architecture with three parallel branches. More precisely, ACM is a channel attention-based module that extracts weighted features from RGB and depth branches. The architecture preserves the inference of the original RGB and depth branches, and enables the fusion branch at the same time. Based on the above structures, ACNet is capable of exploiting more high-quality features from different channels. We evaluate our model on SUN-RGBD and NYUDv2 datasets, and prove that our model outperforms state-of-the-art methods. In particular, a mIoU score of 48.3% on NYUDv2 test set is achieved with ResNet50. We will release our source code based on PyTorch and the trained segmentation model at https://github.com/anheidelonghu/ACNet.