# Pose2Instance: Harnessing Keypoints for Person Instance Segmentation

arXiv: https://arxiv.org/abs/1704.01152 · HTML: https://mlsanity.com/paper/1704.01152/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Apr 2017.

## Built on by (most cited first, 7 of 7)

- [Hand Image Understanding via Deep Multi-Task Learning](https://mlsanity.com/paper/2107.11646/) (2021)
- [Improving Few-Shot Part Segmentation using Coarse Supervision](https://mlsanity.com/paper/2204.05393/) (2022)
- [Test-time Adaptation vs. Training-time Generalization: A Case Study in Human Instance Segmentation using Keypoints Estimation](https://mlsanity.com/paper/2212.06242/) (2022)
- [Detection, Pose Estimation and Segmentation for Multiple Bodies: Closing the Virtuous Circle](https://mlsanity.com/paper/2412.01562/) (2024)
- [Real-time Human-Centric Segmentation for Complex Video Scenes](https://mlsanity.com/paper/2108.07199/) (2021)
- [SAM-pose2seg: Pose-Guided Human Instance Segmentation in Crowds](https://mlsanity.com/paper/2601.08982/) (2026)
- [BBoxMaskPose v2: Expanding Mutual Conditioning to 3D](https://mlsanity.com/paper/2601.15200/) (2026)

## Abstract (arXiv)

Human keypoints are a well-studied representation of people.We explore how to use keypoint models to improve instance-level person segmentation. The main idea is to harness the notion of a distance transform of oracle provided keypoints or estimated keypoint heatmaps as a prior for person instance segmentation task within a deep neural network. For training and evaluation, we consider all those images from COCO where both instance segmentation and human keypoints annotations are available. We first show how oracle keypoints can boost the performance of existing human segmentation model during inference without any training. Next, we propose a framework to directly learn a deep instance segmentation model conditioned on human pose. Experimental results show that at various Intersection Over Union (IOU) thresholds, in a constrained environment with oracle keypoints, the instance segmentation accuracy achieves 10% to 12% relative improvements over a strong baseline of oracle bounding boxes. In a more realistic environment, without the oracle keypoints, the proposed deep person instance segmentation model conditioned on human pose achieves 3.8% to 10.5% relative improvements comparing with its strongest baseline of a deep network trained only for segmentation.