# Learning Rich Features from RGB-D Images for Object Detection and Segmentation

arXiv: https://arxiv.org/abs/1407.5736 · HTML: https://mlsanity.com/paper/1407.5736/

## Key facts

- Cited by 127 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Jul 2014.

## Built on by (most cited first, 30 of 127)

- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving](https://mlsanity.com/paper/2304.14365/) (2023)
- [CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers](https://mlsanity.com/paper/2203.04838/) (2022)
- [Pixel Difference Networks for Efficient Edge Detection](https://mlsanity.com/paper/2108.07009/) (2021)
- [Deep Parametric Continuous Convolutional Neural Networks](https://mlsanity.com/paper/2101.06742/) (2021)
- [ShapeLLM: Universal 3D Object Understanding for Embodied Interaction](https://mlsanity.com/paper/2402.17766/) (2024)
- [Delivering Arbitrary-Modal Semantic Segmentation](https://mlsanity.com/paper/2303.01480/) (2023)
- [Bidirectional Projection Network for Cross Dimension Scene Understanding](https://mlsanity.com/paper/2103.14326/) (2021)
- [ShapeConv: Shape-aware Convolutional Layer for Indoor RGB-D Semantic Segmentation](https://mlsanity.com/paper/2108.10528/) (2021)
- [3D Semantic Scene Completion: a Survey](https://mlsanity.com/paper/2103.07466/) (2021)
- [Discrete Cosine Transform Network for Guided Depth Map Super-Resolution](https://mlsanity.com/paper/2104.06977/) (2021)
- [Exploring Relational Context for Multi-Task Dense Prediction](https://mlsanity.com/paper/2104.13874/) (2021)
- [Semantics for Robotic Mapping, Perception and Interaction: A Survey](https://mlsanity.com/paper/2101.00443/) (2021)
- [Generative Zero-Shot Learning for Semantic Segmentation of 3D Point Clouds](https://mlsanity.com/paper/2108.06230/) (2021)
- [Virtual Normal: Enforcing Geometric Constraints for Accurate and Robust Depth Prediction](https://mlsanity.com/paper/2103.04216/) (2021)
- [3D-to-2D Distillation for Indoor Scene Parsing](https://mlsanity.com/paper/2104.02243/) (2021)
- [Exploring Adversarial Robustness of Multi-Sensor Perception Systems in Self Driving](https://mlsanity.com/paper/2101.06784/) (2021)
- [4D-Net for Learned Multi-Modal Alignment](https://mlsanity.com/paper/2109.01066/) (2021)
- [GeminiFusion: Efficient Pixel-wise Multimodal Fusion for Vision Transformer](https://mlsanity.com/paper/2406.01210/) (2024)
- [Deep Reinforcement Learning in Computer Vision: A Comprehensive Survey](https://mlsanity.com/paper/2108.11510/) (2021)
- [Depth Completion with Twin Surface Extrapolation at Occlusion Boundaries](https://mlsanity.com/paper/2104.02253/) (2021)
- [On Model Calibration for Long-Tailed Object Detection and Instance Segmentation](https://mlsanity.com/paper/2107.02170/) (2021)
- [CIR-Net: Cross-modality Interaction and Refinement for RGB-D Salient Object Detection](https://mlsanity.com/paper/2210.02843/) (2022)
- [EDTER: Edge Detection with Transformer](https://mlsanity.com/paper/2203.08566/) (2022)
- [Decoupling Common and Unique Representations for Multimodal Self-supervised Learning](https://mlsanity.com/paper/2309.05300/) (2023)
- [Cross-modality Discrepant Interaction Network for RGB-D Salient Object Detection](https://mlsanity.com/paper/2108.01971/) (2021)
- [SOSD-Net: Joint Semantic Object Segmentation and Depth Estimation from Monocular images](https://mlsanity.com/paper/2101.07422/) (2021)
- [ProtoSeg: Interpretable Semantic Segmentation with Prototypical Parts](https://mlsanity.com/paper/2301.12276/) (2023)
- [Joint Implicit Image Function for Guided Depth Super-Resolution](https://mlsanity.com/paper/2107.08717/) (2021)
- [Deep Learning Based 3D Segmentation: A Survey](https://mlsanity.com/paper/2103.05423/) (2021)

## Abstract (arXiv)

In this paper we study the problem of object detection for RGB-D images using semantically rich image and depth features. We propose a new geocentric embedding for depth images that encodes height above ground and angle with gravity for each pixel in addition to the horizontal disparity. We demonstrate that this geocentric embedding works better than using raw depth images for learning feature representations with convolutional neural networks. Our final object detection system achieves an average precision of 37.3%, which is a 56% relative improvement over existing methods. We then focus on the task of instance segmentation where we label pixels belonging to object instances found by our detector. For this task, we propose a decision forest approach that classifies pixels in the detection window as foreground or background using a family of unary and binary tests that query shape and geocentric pose features. Finally, we use the output from our object detectors in an existing superpixel classification framework for semantic scene segmentation and achieve a 24% relative improvement over current state-of-the-art for the object categories that we study. We believe advances such as those represented in this paper will facilitate the use of perception in fields like robotics.