# RGB-D Object Detection and Semantic Segmentation for Autonomous Manipulation in Clutter

arXiv: https://arxiv.org/abs/1810.00818 · HTML: https://mlsanity.com/paper/1810.00818/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Oct 2018.

## Built on by (most cited first, 21 of 21)

- [A Survey of Embodied Learning for Object-Centric Robotic Manipulation](https://mlsanity.com/paper/2408.11537/) (2024)
- [SuctionNet-1Billion: A Large-Scale Benchmark for Suction Grasping](https://mlsanity.com/paper/2103.12311/) (2021)
- [From SLAM to Situational Awareness: Challenges and Survey](https://mlsanity.com/paper/2110.00273/) (2021)
- [Contrastive Learning for Enhancing Robust Scene Transfer in Vision-based Agile Flight](https://mlsanity.com/paper/2309.09865/) (2023)
- [The Implicit Values of A Good Hand Shake: Handheld Multi-Frame Neural Depth Refinement](https://mlsanity.com/paper/2111.13738/) (2021)
- [Single View Geocentric Pose in the Wild](https://mlsanity.com/paper/2105.08229/) (2021)
- [Robust Double-Encoder Network for RGB-D Panoptic Segmentation](https://mlsanity.com/paper/2210.02834/) (2022)
- [Shakes on a Plane: Unsupervised Depth Estimation from Unstabilized Photography](https://mlsanity.com/paper/2212.12324/) (2022)
- [Scene Editing as Teleoperation: A Case Study in 6DoF Kit Assembly](https://mlsanity.com/paper/2110.04450/) (2021)
- [Towards Robust Part-aware Instance Segmentation for Industrial Bin Picking](https://mlsanity.com/paper/2203.02767/) (2022)
- [Smart Explorer: Recognizing Objects in Dense Clutter via Interactive Exploration](https://mlsanity.com/paper/2208.03496/) (2022)
- [PIG-Net: Inception based Deep Learning Architecture for 3D Point Cloud Segmentation](https://mlsanity.com/paper/2101.11987/) (2021)
- [Learning Implicit Probability Distribution Functions for Symmetric Orientation Estimation from RGB Images Without Pose Labels](https://mlsanity.com/paper/2211.11394/) (2022)
- [PM-DETR: Domain Adaptive Prompt Memory for Object Detection with Transformers](https://mlsanity.com/paper/2307.00313/) (2023)
- [Object-Oriented Material Classification and 3D Clustering for Improved Semantic Perception and Mapping in Mobile Robots](https://mlsanity.com/paper/2407.06077/) (2024)
- [Open-World Panoptic Segmentation](https://mlsanity.com/paper/2412.12740/) (2024)
- [Category-level Shape Estimation for Densely Cluttered Objects](https://mlsanity.com/paper/2302.11983/) (2023)
- [Clothes Grasping and Unfolding Based on RGB-D Semantic Segmentation](https://mlsanity.com/paper/2305.03259/) (2023)
- [FusionVision: A comprehensive approach of 3D object reconstruction and segmentation from RGB-D cameras using YOLO and fast segment anything](https://mlsanity.com/paper/2403.00175/) (2024)
- [Fusing RGBD Tracking and Segmentation Tree Sampling for Multi-Hypothesis Volumetric Segmentation](https://mlsanity.com/paper/2104.00205/) (2021)
- [Robust Scene Transfer for PointGoal Navigation via Privileged Sensor Guided Contrastive Learning](https://mlsanity.com/paper/2606.05506/) (2026)

## Abstract (arXiv)

Autonomous robotic manipulation in clutter is challenging. A large variety of objects must be perceived in complex scenes, where they are partially occluded and embedded among many distractors, often in restricted spaces. To tackle these challenges, we developed a deep-learning approach that combines object detection and semantic segmentation. The manipulation scenes are captured with RGB-D cameras, for which we developed a depth fusion method. Employing pretrained features makes learning from small annotated robotic data sets possible. We evaluate our approach on two challenging data sets: one captured for the Amazon Picking Challenge 2016, where our team NimbRo came in second in the Stowing and third in the Picking task, and one captured in disaster-response scenarios. The experiments show that object detection and semantic segmentation complement each other and can be combined to yield reliable object perception.