# Differentiable Multi-Granularity Human Representation Learning for Instance-Aware Human Semantic Parsing

arXiv: https://arxiv.org/abs/2103.04570 · HTML: https://mlsanity.com/paper/2103.04570/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 69 references that are in the map.
- First posted on arXiv 8 Mar 2021.

## Builds on (most cited first, 30 of 49)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [DensePose: Dense Human Pose Estimation In The Wild](https://mlsanity.com/paper/1802.00434/) (2018)
- [Cascaded Pyramid Network for Multi-Person Pose Estimation](https://mlsanity.com/paper/1711.07319/) (2017)
- [RMPE: Regional Multi-person Pose Estimation](https://mlsanity.com/paper/1612.00137/) (2016)
- [RepPoints: Point Set Representation for Object Detection](https://mlsanity.com/paper/1904.11490/) (2019)
- [Associative Embedding: End-to-End Learning for Joint Detection and Grouping](https://mlsanity.com/paper/1611.05424/) (2016)
- [PolarMask: Single Shot Instance Segmentation with Polar Representation](https://mlsanity.com/paper/1909.13226/) (2019)
- [Learning Human-Object Interactions by Graph Parsing Neural Networks](https://mlsanity.com/paper/1808.07962/) (2018)
- [Human-Aware Motion Deblurring](https://mlsanity.com/paper/2001.06816/) (2020)
- [Simultaneous Detection and Segmentation](https://mlsanity.com/paper/1407.1808/) (2014)
- [Instance-level Human Parsing via Part Grouping Network](https://mlsanity.com/paper/1808.00157/) (2018)
- [PersonLab: Person Pose Estimation and Instance Segmentation with a Bottom-Up, Part-Based, Geometric Embedding Model](https://mlsanity.com/paper/1803.08225/) (2018)
- [Towards Accurate Multi-person Pose Estimation in the Wild](https://mlsanity.com/paper/1701.01779/) (2017)
- [Iterative Residual Refinement for Joint Optical Flow and Occlusion Estimation](https://mlsanity.com/paper/1904.05290/) (2019)
- [Instance-aware Semantic Segmentation via Multi-task Network Cascades](https://mlsanity.com/paper/1512.04412/) (2015)
- [PifPaf: Composite Fields for Human Pose Estimation](https://mlsanity.com/paper/1903.06593/) (2019)
- [CARAFE: Content-Aware ReAssembly of FEatures](https://mlsanity.com/paper/1905.02188/) (2019)
- [Look into Person: Self-supervised Structure-sensitive Learning and A New Benchmark for Human Parsing](https://mlsanity.com/paper/1703.05446/) (2017)
- [Pose2Seg: Detection Free Human Instance Segmentation](https://mlsanity.com/paper/1803.10683/) (2018)
- [TensorMask: A Foundation for Dense Object Segmentation](https://mlsanity.com/paper/1903.12174/) (2019)
- [DeeperCut: A Deeper, Stronger, and Faster Multi-Person Pose Estimation Model](https://mlsanity.com/paper/1605.03170/) (2016)
- [Single-Stage Multi-Person Pose Machines](https://mlsanity.com/paper/1908.09220/) (2019)

## Built on by (most cited first, 30 of 31)

- [Deep Hierarchical Semantic Segmentation](https://mlsanity.com/paper/2203.14335/) (2022)
- [Regional Semantic Contrast and Aggregation for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2203.09653/) (2022)
- [Going Denser with Open-Vocabulary Part Segmentation](https://mlsanity.com/paper/2305.11173/) (2023)
- [Panoptic-PartFormer: Learning a Unified Model for Panoptic Part Segmentation](https://mlsanity.com/paper/2204.04655/) (2022)
- [Few-shot Segmentation with Optimal Transport Matching and Message Flow](https://mlsanity.com/paper/2108.08518/) (2021)
- [CLUSTSEG: Clustering for Universal Segmentation](https://mlsanity.com/paper/2305.02187/) (2023)
- [OV-PARTS: Towards Open-Vocabulary Part Segmentation](https://mlsanity.com/paper/2310.05107/) (2023)
- [MMR: A Large-scale Benchmark Dataset for Multi-target and Multi-granularity Reasoning Segmentation](https://mlsanity.com/paper/2503.13881/) (2025)
- [Target-Aware Object Discovery and Association for Unsupervised Video Multi-Object Segmentation](https://mlsanity.com/paper/2104.04782/) (2021)
- [PanopticPartFormer++: A Unified and Decoupled View for Panoptic Part Segmentation](https://mlsanity.com/paper/2301.00954/) (2023)
- [Weakly-Supervised Audio-Visual Segmentation](https://mlsanity.com/paper/2311.15080/) (2023)
- [Deep Learning Technique for Human Parsing: A Survey and Outlook](https://mlsanity.com/paper/2301.00394/) (2023)
- [Learning Equivariant Segmentation with Instance-Unique Querying](https://mlsanity.com/paper/2210.00911/) (2022)
- [Fashionformer: A simple, Effective and Unified Baseline for Human Fashion Segmentation and Recognition](https://mlsanity.com/paper/2204.04654/) (2022)
- [PartGLEE: A Foundation Model for Recognizing and Parsing Any Objects](https://mlsanity.com/paper/2407.16696/) (2024)
- [AIParsing: Anchor-free Instance-level Human Parsing](https://mlsanity.com/paper/2207.06854/) (2022)
- [Nonverbal Interaction Detection](https://mlsanity.com/paper/2407.08133/) (2024)
- [Quality-Aware Network for Human Parsing](https://mlsanity.com/paper/2103.05997/) (2021)
- [Multi-Granularity Regularized Re-Balancing for Class Incremental Learning](https://mlsanity.com/paper/2206.15189/) (2022)
- [PointScatter: Point Set Representation for Tubular Structure Extraction](https://mlsanity.com/paper/2209.05774/) (2022)
- [Robust Backdoor Attacks on Object Detection in Real World](https://mlsanity.com/paper/2309.08953/) (2023)
- [CALICO: Part-Focused Semantic Co-Segmentation with Large Vision-Language Models](https://mlsanity.com/paper/2412.19331/) (2024)
- [UniParser: Multi-Human Parsing with Unified Correlation Representation Learning](https://mlsanity.com/paper/2310.08984/) (2023)
- [Dense Center-Direction Regression for Object Counting and Localization with Point Supervision](https://mlsanity.com/paper/2408.14457/) (2024)
- [Parsing Objects at a Finer Granularity: A Survey](https://mlsanity.com/paper/2212.13693/) (2022)
- [Single-stage Multi-human Parsing via Point Sets and Center-based Offsets](https://mlsanity.com/paper/2304.11356/) (2023)
- [Deep Learning for Human Parsing: A Survey](https://mlsanity.com/paper/2301.12416/) (2023)
- [Feature point detection in HDR images based on coefficient of variation](https://mlsanity.com/paper/2304.10666/) (2023)
- [Exploring the Robustness of Human Parsers Towards Common Corruptions](https://mlsanity.com/paper/2309.00938/) (2023)
- [Adjustable Method Based on Body Parts for Improving the Accuracy of 3D Reconstruction in Visually Important Body Parts from Silhouettes](https://mlsanity.com/paper/2211.14822/) (2022)

## Abstract (arXiv)

To address the challenging task of instance-aware human part parsing, a new bottom-up regime is proposed to learn category-level human semantic segmentation as well as multi-person pose estimation in a joint and end-to-end manner. It is a compact, efficient and powerful framework that exploits structural information over different human granularities and eases the difficulty of person partitioning. Specifically, a dense-to-sparse projection field, which allows explicitly associating dense human semantics with sparse keypoints, is learnt and progressively improved over the network feature pyramid for robustness. Then, the difficult pixel grouping problem is cast as an easier, multi-person joint assembling task. By formulating joint association as maximum-weight bipartite matching, a differentiable solution is developed to exploit projected gradient descent and Dykstra's cyclic projection algorithm. This makes our method end-to-end trainable and allows back-propagating the grouping error to directly supervise multi-granularity human representation learning. This is distinguished from current bottom-up human parsers or pose estimators which require sophisticated post-processing or heuristic greedy algorithms. Experiments on three instance-aware human parsing datasets show that our model outperforms other bottom-up alternatives with much more efficient inference.