# SAM-pose2seg: Pose-Guided Human Instance Segmentation in Crowds

arXiv: https://arxiv.org/abs/2601.08982 · HTML: https://mlsanity.com/paper/2601.08982/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 33 references that are in the map.
- First posted on arXiv 13 Jan 2026.

## Builds on (most cited first, 27 of 27)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [YOLOX: Exceeding YOLO Series in 2021](https://mlsanity.com/paper/2107.08430/) (2021)
- [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://mlsanity.com/paper/2401.14159/) (2024)
- [Exploring Plain Vision Transformer Backbones for Object Detection](https://mlsanity.com/paper/2203.16527/) (2022)
- [CrowdHuman: A Benchmark for Detecting Human in a Crowd](https://mlsanity.com/paper/1805.00123/) (2018)
- [DETRs with Collaborative Hybrid Assignments Training](https://mlsanity.com/paper/2211.12860/) (2022)
- [CrowdPose: Efficient Crowded Scenes Pose Estimation and A New Benchmark](https://mlsanity.com/paper/1812.00324/) (2018)
- [RTMDet: An Empirical Study of Designing Real-Time Object Detectors](https://mlsanity.com/paper/2212.07784/) (2022)
- [Instance-level Human Parsing via Part Grouping Network](https://mlsanity.com/paper/1808.00157/) (2018)
- [Pose2Seg: Detection Free Human Instance Segmentation](https://mlsanity.com/paper/1803.10683/) (2018)
- [AI Challenger : A Large-scale Dataset for Going Deeper in Image Understanding](https://mlsanity.com/paper/1711.06475/) (2017)
- [Parsing R-CNN for Instance-Level Human Analysis](https://mlsanity.com/paper/1811.12596/) (2018)
- [SAMWISE: Infusing Wisdom in SAM2 for Text-Driven Video Segmentation](https://mlsanity.com/paper/2411.17646/) (2024)
- [Semantic-aware SAM for Point-Prompted Instance Segmentation](https://mlsanity.com/paper/2312.15895/) (2023)
- [OpenWorldSAM: Extending SAM2 for Universal Image Segmentation with Language Prompts](https://mlsanity.com/paper/2507.05427/) (2025)
- [Crowd-SAM: SAM as a Smart Annotator for Object Detection in Crowded Scenes](https://mlsanity.com/paper/2407.11464/) (2024)
- [IterDet: Iterative Scheme for Object Detection in Crowded Environments](https://mlsanity.com/paper/2005.05708/) (2020)
- [ProbPose: A Probabilistic Approach to 2D Human Pose Estimation](https://mlsanity.com/paper/2412.02254/) (2024)
- [Pose2Instance: Harnessing Keypoints for Person Instance Segmentation](https://mlsanity.com/paper/1704.01152/) (2017)
- [Test-time Adaptation vs. Training-time Generalization: A Case Study in Human Instance Segmentation using Keypoints Estimation](https://mlsanity.com/paper/2212.06242/) (2022)
- [Detection, Pose Estimation and Segmentation for Multiple Bodies: Closing the Virtuous Circle](https://mlsanity.com/paper/2412.01562/) (2024)

## Built on by (most cited first, 1 of 1)

- [BBoxMaskPose v2: Expanding Mutual Conditioning to 3D](https://mlsanity.com/paper/2601.15200/) (2026)

## Abstract (arXiv)

Segment Anything (SAM) provides an unprecedented foundation for human segmentation, but may struggle under occlusion, where keypoints may be partially or fully invisible. We adapt SAM 2.1 for pose-guided segmentation with minimal encoder modifications, retaining its strong generalization. Using a fine-tuning strategy called PoseMaskRefine, we incorporate pose keypoints with high visibility into the iterative correction process originally employed by SAM, yielding improved robustness and accuracy across multiple datasets. During inference, we simplify prompting by selecting only the three keypoints with the highest visibility. This strategy reduces sensitivity to common errors, such as missing body parts or misclassified clothing, and allows accurate mask prediction from as few as a single keypoint. Our results demonstrate that pose-guided fine-tuning of SAM enables effective, occlusion-aware human segmentation while preserving the generalization capabilities of the original model. The code and pretrained models will be available at https://mirapurkrabek.github.io/BBox-Mask-Pose/.