# Depth-Adapted CNNs for RGB-D Semantic Segmentation

arXiv: https://arxiv.org/abs/2206.03939 · HTML: https://mlsanity.com/paper/2206.03939/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 63 references that are in the map.
- First posted on arXiv 8 Jun 2022.

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [KPConv: Flexible and Deformable Convolution for Point Clouds](https://mlsanity.com/paper/1904.08889/) (2019)
- [Multi-Scale Context Aggregation by Dilated Convolutions](https://mlsanity.com/paper/1511.07122/) (2015)
- [PointCNN: Convolution On X-Transformed Points](https://mlsanity.com/paper/1801.07791/) (2018)
- [Multi-View 3D Object Detection Network for Autonomous Driving](https://mlsanity.com/paper/1611.07759/) (2016)
- [Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture](https://mlsanity.com/paper/1411.4734/) (2014)
- [Semantic Scene Completion from a Single Depth Image](https://mlsanity.com/paper/1611.08974/) (2016)
- [Spherical CNNs](https://mlsanity.com/paper/1801.10130/) (2018)
- [Conditional Positional Encodings for Vision Transformers](https://mlsanity.com/paper/2102.10882/) (2021)
- [Relation-Shape Convolutional Neural Network for Point Cloud Analysis](https://mlsanity.com/paper/1904.07601/) (2019)
- [Volumetric and Multi-View CNNs for Object Classification on 3D Data](https://mlsanity.com/paper/1604.03265/) (2016)
- [SEGCloud: Semantic Segmentation of 3D Point Clouds](https://mlsanity.com/paper/1710.07563/) (2017)
- [Learning Rich Features from RGB-D Images for Object Detection and Segmentation](https://mlsanity.com/paper/1407.5736/) (2014)
- [Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentation](https://mlsanity.com/paper/2007.09183/) (2020)
- [Vehicle Detection from 3D Lidar Using Fully Convolutional Network](https://mlsanity.com/paper/1608.07916/) (2016)
- [ACNet: Attention Based Network to Exploit Complementary Features for RGBD Semantic Segmentation](https://mlsanity.com/paper/1905.10089/) (2019)
- [DensePoint: Learning Densely Contextual Representation for Efficient Point Cloud Processing](https://mlsanity.com/paper/1909.03669/) (2019)
- [Efficient RGB-D Semantic Segmentation for Indoor Scene Analysis](https://mlsanity.com/paper/2011.06961/) (2020)

## Built on by (most cited first, 6 of 6)

- [Segment Anything in 3D with Radiance Fields](https://mlsanity.com/paper/2304.12308/) (2023)
- [DFormer: Rethinking RGBD Representation Learning for Semantic Segmentation](https://mlsanity.com/paper/2309.09668/) (2023)
- [Pixel Difference Convolutional Network for RGB-D Semantic Segmentation](https://mlsanity.com/paper/2302.11951/) (2023)
- [Robust RGB-D Fusion for Saliency Detection](https://mlsanity.com/paper/2208.01762/) (2022)
- [Toward Efficient Visual Gyroscopes: Spherical Moments, Harmonics Filtering, and Masking Techniques for Spherical Camera Applications](https://mlsanity.com/paper/2404.01924/) (2024)
- [Efficient RGB-D Scene Understanding via Multi-task Adaptive Learning and Cross-dimensional Feature Guidance](https://mlsanity.com/paper/2603.07570/) (2026)

## Abstract (arXiv)

Recent RGB-D semantic segmentation has motivated research interest thanks to the accessibility of complementary modalities from the input side. Existing works often adopt a two-stream architecture that processes photometric and geometric information in parallel, with few methods explicitly leveraging the contribution of depth cues to adjust the sampling position on RGB images. In this paper, we propose a novel framework to incorporate the depth information in the RGB convolutional neural network (CNN), termed Z-ACN (Depth-Adapted CNN). Specifically, our Z-ACN generates a 2D depth-adapted offset which is fully constrained by low-level features to guide the feature extraction on RGB images. With the generated offset, we introduce two intuitive and effective operations to replace basic CNN operators: depth-adapted convolution and depth-adapted average pooling. Extensive experiments on both indoor and outdoor semantic segmentation tasks demonstrate the effectiveness of our approach.