# Recurrent Pixel Embedding for Instance Grouping

arXiv: https://arxiv.org/abs/1712.08273 · HTML: https://mlsanity.com/paper/1712.08273/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Dec 2017.

## Built on by (most cited first, 30 of 58)

- [Exploring Cross-Image Pixel Contrast for Semantic Segmentation](https://mlsanity.com/paper/2101.11939/) (2021)
- [Unsupervised Semantic Segmentation by Contrasting Object Mask Proposals](https://mlsanity.com/paper/2102.06191/) (2021)
- [Rethinking Semantic Segmentation: A Prototype View](https://mlsanity.com/paper/2203.15102/) (2022)
- [Hierarchical Aggregation for 3D Instance Segmentation](https://mlsanity.com/paper/2108.02350/) (2021)
- [GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinement](https://mlsanity.com/paper/2104.09958/) (2021)
- [Deep Occlusion-Aware Instance Segmentation with Overlapping BiLayers](https://mlsanity.com/paper/2103.12340/) (2021)
- [Contrastive Lift: 3D Object Instance Segmentation by Slow-Fast Contrastive Fusion](https://mlsanity.com/paper/2306.04633/) (2023)
- [Incremental Few-Shot Instance Segmentation](https://mlsanity.com/paper/2105.05312/) (2021)
- [Spatial Feature Calibration and Temporal Fusion for Effective One-stage Video Instance Segmentation](https://mlsanity.com/paper/2104.05606/) (2021)
- [Contrastive Mean-Shift Learning for Generalized Category Discovery](https://mlsanity.com/paper/2404.09451/) (2024)
- [Look Before You Match: Instance Understanding Matters in Video Object Segmentation](https://mlsanity.com/paper/2212.06826/) (2022)
- [CLUSTSEG: Clustering for Universal Segmentation](https://mlsanity.com/paper/2305.02187/) (2023)
- [A Weakly Supervised Amodal Segmenter with Boundary Uncertainty Estimation](https://mlsanity.com/paper/2108.09897/) (2021)
- [Mean Shift Mask Transformer for Unseen Object Instance Segmentation](https://mlsanity.com/paper/2211.11679/) (2022)
- [Learning Equivariant Segmentation with Instance-Unique Querying](https://mlsanity.com/paper/2210.00911/) (2022)
- [FAPIS: A Few-shot Anchor-free Part-based Instance Segmenter](https://mlsanity.com/paper/2104.00073/) (2021)
- [RICE: Refining Instance Masks in Cluttered Environments with Graph Neural Networks](https://mlsanity.com/paper/2106.15711/) (2021)
- [Towards Neural Diarization for Unlimited Numbers of Speakers Using Global and Local Attractors](https://mlsanity.com/paper/2107.01545/) (2021)
- [DynaMITe: Dynamic Query Bootstrapping for Multi-object Interactive Segmentation Transformer](https://mlsanity.com/paper/2304.06668/) (2023)
- [Geometrically-driven Aggregation for Zero-shot 3D Point Cloud Understanding](https://mlsanity.com/paper/2312.02244/) (2023)
- [iShape: A First Step Towards Irregular Shape Instance Segmentation](https://mlsanity.com/paper/2109.15068/) (2021)
- [3D Instance Segmentation of MVS Buildings](https://mlsanity.com/paper/2112.09902/) (2021)
- [NeuralBF: Neural Bilateral Filtering for Top-down Instance Segmentation on Point Clouds](https://mlsanity.com/paper/2207.09978/) (2022)
- [Clustering Propagation for Universal Medical Image Segmentation](https://mlsanity.com/paper/2403.16646/) (2024)
- [DeepSportLab: a Unified Framework for Ball Detection, Player Instance Segmentation and Pose Estimation in Team Sports Scenes](https://mlsanity.com/paper/2112.00627/) (2021)
- [LiDAR-based 4D Panoptic Segmentation via Dynamic Shifting Network](https://mlsanity.com/paper/2203.07186/) (2022)
- [Indoor Panorama Planar 3D Reconstruction via Divide and Conquer](https://mlsanity.com/paper/2106.14166/) (2021)
- [You Only Learn One Query: Learning Unified Human Query for Single-Stage Multi-Person Multi-Task Human-Centric Perception](https://mlsanity.com/paper/2312.05525/) (2023)
- [Online Neural Diarization of Unlimited Numbers of Speakers Using Global and Local Attractors](https://mlsanity.com/paper/2206.02432/) (2022)
- [Open-world Instance Segmentation: Top-down Learning with Bottom-up Supervision](https://mlsanity.com/paper/2303.05503/) (2023)

## Abstract (arXiv)

We introduce a differentiable, end-to-end trainable framework for solving pixel-level grouping problems such as instance segmentation consisting of two novel components. First, we regress pixels into a hyper-spherical embedding space so that pixels from the same group have high cosine similarity while those from different groups have similarity below a specified margin. We analyze the choice of embedding dimension and margin, relating them to theoretical results on the problem of distributing points uniformly on the sphere. Second, to group instances, we utilize a variant of mean-shift clustering, implemented as a recurrent neural network parameterized by kernel bandwidth. This recurrent grouping module is differentiable, enjoys convergent dynamics and probabilistic interpretability. Backpropagating the group-weighted loss through this module allows learning to focus on only correcting embedding errors that won't be resolved during subsequent clustering. Our framework, while conceptually simple and theoretically abundant, is also practically effective and computationally efficient. We demonstrate substantial improvements over state-of-the-art instance segmentation for object proposal generation, as well as demonstrating the benefits of grouping loss for classification tasks such as boundary detection and semantic segmentation.