# Fusing RGBD Tracking and Segmentation Tree Sampling for Multi-Hypothesis Volumetric Segmentation

arXiv: https://arxiv.org/abs/2104.00205 · HTML: https://mlsanity.com/paper/2104.00205/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 43 references that are in the map.
- First posted on arXiv 1 Apr 2021.

## Builds on (most cited first, 26 of 26)

- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [OctNet: Learning Deep 3D Representations at High Resolutions](https://mlsanity.com/paper/1611.05009/) (2016)
- [SOLO: Segmenting Objects by Locations](https://mlsanity.com/paper/1912.04488/) (2019)
- [Fast Online Object Tracking and Segmentation: A Unifying Approach](https://mlsanity.com/paper/1812.05050/) (2018)
- [Volumetric and Multi-View CNNs for Object Classification on 3D Data](https://mlsanity.com/paper/1604.03265/) (2016)
- [One-Shot Video Object Segmentation](https://mlsanity.com/paper/1611.05198/) (2016)
- [3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans](https://mlsanity.com/paper/1812.07003/) (2018)
- [Exploring Uncertainty Measures in Deep Networks for Multiple Sclerosis Lesion Detection and Segmentation](https://mlsanity.com/paper/1808.01200/) (2018)
- [Interactive Perception: Leveraging Action in Perception and Perception in Action](https://mlsanity.com/paper/1604.03670/) (2016)
- [Multi-view Self-supervised Deep Learning for 6D Pose Estimation in the Amazon Picking Challenge](https://mlsanity.com/paper/1609.09475/) (2016)
- [MID-Fusion: Octree-based Object-Level Multi-Instance Dynamic SLAM](https://mlsanity.com/paper/1812.07976/) (2018)
- [VV-Net: Voxel VAE Net with Group Convolutions for Point Cloud Segmentation](https://mlsanity.com/paper/1811.04337/) (2018)
- [Factoring Shape, Pose, and Layout from the 2D Image of a 3D Scene](https://mlsanity.com/paper/1712.01812/) (2017)
- [The Best of Both Modes: Separately Leveraging RGB and Depth for Unseen Object Instance Segmentation](https://mlsanity.com/paper/1907.13236/) (2019)
- [ForkNet: Multi-branch Volumetric Semantic Completion from a Single Depth Image](https://mlsanity.com/paper/1909.01106/) (2019)
- [Convolutional Oriented Boundaries](https://mlsanity.com/paper/1608.02755/) (2016)
- [RGB-D Object Detection and Semantic Segmentation for Autonomous Manipulation in Clutter](https://mlsanity.com/paper/1810.00818/) (2018)
- [Object Finding in Cluttered Scenes Using Interactive Perception](https://mlsanity.com/paper/1911.07482/) (2019)
- [Robust Grasp Planning Over Uncertain Shape Completions](https://mlsanity.com/paper/1903.00645/) (2019)
- [SceneCut: Joint Geometric and Object Segmentation for Indoor Scenes](https://mlsanity.com/paper/1709.07158/) (2017)
- [The Manifold Particle Filter for State Estimation on High-dimensional Implicit Manifolds](https://mlsanity.com/paper/1604.07224/) (2016)
- [SUM: Sequential Scene Understanding and Manipulation](https://mlsanity.com/paper/1703.07491/) (2017)
- [Inferring Occluded Geometry Improves Performance when Retrieving an Object from Dense Clutter](https://mlsanity.com/paper/1907.08770/) (2019)
- [Tree-Cut for Probabilistic Image Segmentation](https://mlsanity.com/paper/1506.03852/) (2015)
- [Kinematically-Informed Interactive Perception: Robot-Generated 3D Models for Classification](https://mlsanity.com/paper/1901.05580/) (2019)

## Abstract (arXiv)

Despite rapid progress in scene segmentation in recent years, 3D segmentation methods are still limited when there is severe occlusion. The key challenge is estimating the segment boundaries of (partially) occluded objects, which are inherently ambiguous when considering only a single frame. In this work, we propose Multihypothesis Segmentation Tracking (MST), a novel method for volumetric segmentation in changing scenes, which allows scene ambiguity to be tracked and our estimates to be adjusted over time as we interact with the scene. Two main innovations allow us to tackle this difficult problem: 1) A novel way to sample possible segmentations from a segmentation tree; and 2) A novel approach to fusing tracking results with multiple segmentation estimates. These methods allow MST to track the segmentation state over time and incorporate new information, such as new objects being revealed. We evaluate our method on several cluttered tabletop environments in simulation and reality. Our results show that MST outperforms baselines in all tested scenes.