# Instance Segmentation by Jointly Optimizing Spatial Embeddings and Clustering Bandwidth

arXiv: https://arxiv.org/abs/1906.11109 · HTML: https://mlsanity.com/paper/1906.11109/

## Key facts

- Cited by 89 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jun 2019.

## Built on by (most cited first, 30 of 89)

- [K-Net: Towards Unified Image Segmentation](https://mlsanity.com/paper/2106.14855/) (2021)
- [RTMDet: An Empirical Study of Designing Real-Time Object Detectors](https://mlsanity.com/paper/2212.07784/) (2022)
- [Towards Open Vocabulary Learning: A Survey](https://mlsanity.com/paper/2306.15880/) (2023)
- [Rethinking Semantic Segmentation: A Prototype View](https://mlsanity.com/paper/2203.15102/) (2022)
- [Hierarchical Aggregation for 3D Instance Segmentation](https://mlsanity.com/paper/2108.02350/) (2021)
- [Transformer-Based Visual Segmentation: A Survey](https://mlsanity.com/paper/2304.09854/) (2023)
- [P3Depth: Monocular Depth Estimation with a Piecewise Planarity Prior](https://mlsanity.com/paper/2204.02091/) (2022)
- [SG-Net: Spatial Granularity Network for One-Stage Video Instance Segmentation](https://mlsanity.com/paper/2103.10284/) (2021)
- [4D Panoptic LiDAR Segmentation](https://mlsanity.com/paper/2102.12472/) (2021)
- [CMT-DeepLab: Clustering Mask Transformers for Panoptic Segmentation](https://mlsanity.com/paper/2206.08948/) (2022)
- [Look Closer to Segment Better: Boundary Patch Refinement for Instance Segmentation](https://mlsanity.com/paper/2104.05239/) (2021)
- [Spatial Feature Calibration and Temporal Fusion for Effective One-stage Video Instance Segmentation](https://mlsanity.com/paper/2104.05606/) (2021)
- [SOTR: Segmenting Objects with Transformers](https://mlsanity.com/paper/2108.06747/) (2021)
- [DeepSportradar-v1: Computer Vision Dataset for Sports Understanding with High Quality Annotations](https://mlsanity.com/paper/2208.08190/) (2022)
- [4D-StOP: Panoptic Segmentation of 4D LiDAR using Spatio-temporal Object Proposal Generation and Aggregation](https://mlsanity.com/paper/2209.14858/) (2022)
- [Look Before You Match: Instance Understanding Matters in Video Object Segmentation](https://mlsanity.com/paper/2212.06826/) (2022)
- [CLUSTSEG: Clustering for Universal Segmentation](https://mlsanity.com/paper/2305.02187/) (2023)
- [Betrayed by Captions: Joint Caption Grounding and Generation for Open Vocabulary Instance Segmentation](https://mlsanity.com/paper/2301.00805/) (2023)
- [E2EC: An End-to-End Contour-based Method for High-Quality High-Speed Instance Segmentation](https://mlsanity.com/paper/2203.04074/) (2022)
- [Beyond Semantic to Instance Segmentation: Weakly-Supervised Instance Segmentation via Semantic Knowledge Transfer and Self-Refinement](https://mlsanity.com/paper/2109.09477/) (2021)
- [DeepLab2: A TensorFlow Library for Deep Labeling](https://mlsanity.com/paper/2106.09748/) (2021)
- [Plane Geometry Diagram Parsing](https://mlsanity.com/paper/2205.09363/) (2022)
- [Instance and Panoptic Segmentation Using Conditional Convolutions](https://mlsanity.com/paper/2102.03026/) (2021)
- [Panoptic Segmentation: A Review](https://mlsanity.com/paper/2111.10250/) (2021)
- [Learning Equivariant Segmentation with Instance-Unique Querying](https://mlsanity.com/paper/2210.00911/) (2022)
- [Unknown Object Segmentation from Stereo Images](https://mlsanity.com/paper/2103.06796/) (2021)
- [MonoCInIS: Camera Independent Monocular 3D Object Detection using Instance Segmentation](https://mlsanity.com/paper/2110.00464/) (2021)
- [Fashionformer: A simple, Effective and Unified Baseline for Human Fashion Segmentation and Recognition](https://mlsanity.com/paper/2204.04654/) (2022)
- [EmbedTrack -- Simultaneous Cell Segmentation and Tracking Through Learning Offsets and Clustering Bandwidths](https://mlsanity.com/paper/2204.10713/) (2022)
- [PUPS: Point Cloud Unified Panoptic Segmentation](https://mlsanity.com/paper/2302.06185/) (2023)

## Abstract (arXiv)

Current state-of-the-art instance segmentation methods are not suited for real-time applications like autonomous driving, which require fast execution times at high accuracy. Although the currently dominant proposal-based methods have high accuracy, they are slow and generate masks at a fixed and low resolution. Proposal-free methods, by contrast, can generate masks at high resolution and are often faster, but fail to reach the same accuracy as the proposal-based methods. In this work we propose a new clustering loss function for proposal-free instance segmentation. The loss function pulls the spatial embeddings of pixels belonging to the same instance together and jointly learns an instance-specific clustering bandwidth, maximizing the intersection-over-union of the resulting instance mask. When combined with a fast architecture, the network can perform instance segmentation in real-time while maintaining a high accuracy. We evaluate our method on the challenging Cityscapes benchmark and achieve top results (5% improvement over Mask R-CNN) at more than 10 fps on 2MP images. Code will be available at https://github.com/davyneven/SpatialEmbeddings .