# Malleable 2.5D Convolution: Learning Receptive Fields along the Depth-axis for RGB-D Scene Parsing

arXiv: https://arxiv.org/abs/2007.09365 · HTML: https://mlsanity.com/paper/2007.09365/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jul 2020.

## Built on by (most cited first, 24 of 24)

- [CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers](https://mlsanity.com/paper/2203.04838/) (2022)
- [ShapeConv: Shape-aware Convolutional Layer for Indoor RGB-D Semantic Segmentation](https://mlsanity.com/paper/2108.10528/) (2021)
- [OmniSeg3D: Omniversal 3D Segmentation via Hierarchical Contrastive Learning](https://mlsanity.com/paper/2311.11666/) (2023)
- [Open-World Entity Segmentation](https://mlsanity.com/paper/2107.14228/) (2021)
- [Scene-aware Generative Network for Human Motion Synthesis](https://mlsanity.com/paper/2105.14804/) (2021)
- [InverseForm: A Loss Function for Structured Boundary-Aware Segmentation](https://mlsanity.com/paper/2104.02745/) (2021)
- [Segment Anything in 3D with Radiance Fields](https://mlsanity.com/paper/2304.12308/) (2023)
- [A Closer Look at Invariances in Self-supervised Pre-training for 3D Vision](https://mlsanity.com/paper/2207.04997/) (2022)
- [Not All Voxels Are Equal: Semantic Scene Completion from the Point-Voxel Perspective](https://mlsanity.com/paper/2112.12925/) (2021)
- [Pixel Difference Convolutional Network for RGB-D Semantic Segmentation](https://mlsanity.com/paper/2302.11951/) (2023)
- [HS3: Learning with Proper Task Complexity in Hierarchically Supervised Semantic Segmentation](https://mlsanity.com/paper/2111.02333/) (2021)
- [DCANet: Differential Convolution Attention Network for RGB-D Semantic Segmentation](https://mlsanity.com/paper/2210.06747/) (2022)
- [Efficient Multi-Task Scene Analysis with RGB-D Transformers](https://mlsanity.com/paper/2306.05242/) (2023)
- [Attention-based Dual Supervised Decoder for RGBD Semantic Segmentation](https://mlsanity.com/paper/2201.01427/) (2022)
- [Depth-Adapted CNNs for RGB-D Semantic Segmentation](https://mlsanity.com/paper/2206.03939/) (2022)
- [Understanding Dark Scenes by Contrasting Multi-Modal Observations](https://mlsanity.com/paper/2308.12320/) (2023)
- [Split4D: Decomposed 4D Scene Reconstruction Without Video Segmentation](https://mlsanity.com/paper/2512.22745/) (2025)
- [Review on Indoor RGB-D Semantic Segmentation with Deep Convolutional Neural Networks](https://mlsanity.com/paper/2105.11925/) (2021)
- [ComPtr: Towards Diverse Bi-source Dense Prediction Tasks via A Simple yet General Complementary Transformer](https://mlsanity.com/paper/2307.12349/) (2023)
- [Clothes Grasping and Unfolding Based on RGB-D Semantic Segmentation](https://mlsanity.com/paper/2305.03259/) (2023)
- [Efficient Prediction of Dense Visual Embeddings via Distillation and RGB-D Transformers](https://mlsanity.com/paper/2601.00359/) (2026)
- [Project-and-Fuse: Improving RGB-D Semantic Segmentation via Graph Convolution Networks](https://mlsanity.com/paper/2501.18851/) (2025)
- [Efficient RGB-D Scene Understanding via Multi-task Adaptive Learning and Cross-dimensional Feature Guidance](https://mlsanity.com/paper/2603.07570/) (2026)
- [SegRGB-X: General RGB-X Semantic Segmentation Model](https://mlsanity.com/paper/2603.28023/) (2026)

## Abstract (arXiv)

Depth data provide geometric information that can bring progress in RGB-D scene parsing tasks. Several recent works propose RGB-D convolution operators that construct receptive fields along the depth-axis to handle 3D neighborhood relations between pixels. However, these methods pre-define depth receptive fields by hyperparameters, making them rely on parameter selection. In this paper, we propose a novel operator called malleable 2.5D convolution to learn the receptive field along the depth-axis. A malleable 2.5D convolution has one or more 2D convolution kernels. Our method assigns each pixel to one of the kernels or none of them according to their relative depth differences, and the assigning process is formulated as a differentiable form so that it can be learnt by gradient descent. The proposed operator runs on standard 2D feature maps and can be seamlessly incorporated into pre-trained CNNs. We conduct extensive experiments on two challenging RGB-D semantic segmentation dataset NYUDv2 and Cityscapes to validate the effectiveness and the generalization ability of our method.