# Learning Stixel-based Instance Segmentation

arXiv: https://arxiv.org/abs/2107.03070 · HTML: https://mlsanity.com/paper/2107.03070/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 40 references that are in the map.
- First posted on arXiv 7 Jul 2021.

## Builds on (most cited first, 19 of 19)

- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Path Aggregation Network for Instance Segmentation](https://mlsanity.com/paper/1803.01534/) (2018)
- [PointCNN: Convolution On X-Transformed Points](https://mlsanity.com/paper/1801.07791/) (2018)
- [Frustum PointNets for 3D Object Detection from RGB-D Data](https://mlsanity.com/paper/1711.08488/) (2017)
- [OctNet: Learning Deep 3D Representations at High Resolutions](https://mlsanity.com/paper/1611.05009/) (2016)
- [Volumetric and Multi-View CNNs for Object Classification on 3D Data](https://mlsanity.com/paper/1604.03265/) (2016)
- [Speed/accuracy trade-offs for modern convolutional object detectors](https://mlsanity.com/paper/1611.10012/) (2016)
- [SEGCloud: Semantic Segmentation of 3D Point Clouds](https://mlsanity.com/paper/1710.07563/) (2017)
- [UPSNet: A Unified Panoptic Segmentation Network](https://mlsanity.com/paper/1901.03784/) (2019)
- [Learning Object Bounding Boxes for 3D Instance Segmentation on Point Clouds](https://mlsanity.com/paper/1906.01140/) (2019)
- [GSPN: Generative Shape Proposal Network for 3D Instance Segmentation in Point Cloud](https://mlsanity.com/paper/1812.03320/) (2018)
- [Deep Snake for Real-Time Instance Segmentation](https://mlsanity.com/paper/2001.01629/) (2020)
- [Instance Segmentation by Jointly Optimizing Spatial Embeddings and Clustering Bandwidth](https://mlsanity.com/paper/1906.11109/) (2019)
- [The Stixel world: A medium-level representation of traffic scenes](https://mlsanity.com/paper/1704.00280/) (2017)

## Abstract (arXiv)

Stixels have been successfully applied to a wide range of vision tasks in autonomous driving, recently including instance segmentation. However, due to their sparse occurrence in the image, until now Stixels seldomly served as input for Deep Learning algorithms, restricting their utility for such approaches. In this work we present StixelPointNet, a novel method to perform fast instance segmentation directly on Stixels. By regarding the Stixel representation as unstructured data similar to point clouds, architectures like PointNet are able to learn features from Stixels. We use a bounding box detector to propose candidate instances, for which the relevant Stixels are extracted from the input image. On these Stixels, a PointNet models learns binary segmentations, which we then unify throughout the whole image in a final selection step. StixelPointNet achieves state-of-the-art performance on Stixel-level, is considerably faster than pixel-based segmentation methods, and shows that with our approach the Stixel domain can be introduced to many new 3D Deep Learning tasks.