# PanopticPartFormer++: A Unified and Decoupled View for Panoptic Part Segmentation

arXiv: https://arxiv.org/abs/2301.00954 · HTML: https://mlsanity.com/paper/2301.00954/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 95 of its 120 references that are in the map.
- First posted on arXiv 3 Jan 2023.

## Builds on (most cited first, 30 of 95)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018)
- [Deformable ConvNets v2: More Deformable, Better Results](https://mlsanity.com/paper/1811.11168/) (2018)
- [Panoptic Feature Pyramid Networks](https://mlsanity.com/paper/1901.02446/) (2019)
- [Segmentation Transformer: Object-Contextual Representations for Semantic Segmentation](https://mlsanity.com/paper/1909.11065/) (2019)

## Built on by (most cited first, 19 of 19)

- [Towards Open Vocabulary Learning: A Survey](https://mlsanity.com/paper/2306.15880/) (2023)
- [Transformer-Based Visual Segmentation: A Survey](https://mlsanity.com/paper/2304.09854/) (2023)
- [Change Detection Methods for Remote Sensing in the Last Decade: A Comprehensive Review](https://mlsanity.com/paper/2305.05813/) (2023)
- [SLiMe: Segment Like Me](https://mlsanity.com/paper/2309.03179/) (2023)
- [MG-LLaVA: Towards Multi-Granularity Visual Instruction Tuning](https://mlsanity.com/paper/2406.17770/) (2024)
- [Pair then Relation: Pair-Net for Panoptic Scene Graph Generation](https://mlsanity.com/paper/2307.08699/) (2023)
- [DenseWorld-1M: Towards Detailed Dense Grounded Caption in the Real World](https://mlsanity.com/paper/2506.24102/) (2025)
- [OV-VG: A Benchmark for Open-Vocabulary Visual Grounding](https://mlsanity.com/paper/2310.14374/) (2023)
- [SMITE: Segment Me In TimE](https://mlsanity.com/paper/2410.18538/) (2024)
- [AIMS: All-Inclusive Multi-Level Segmentation](https://mlsanity.com/paper/2305.17768/) (2023)
- [RMP-SAM: Towards Real-Time Multi-Purpose Segment Anything](https://mlsanity.com/paper/2401.10228/) (2024)
- [Task-aligned Part-aware Panoptic Segmentation through Joint Object-Part Representations](https://mlsanity.com/paper/2406.10114/) (2024)
- [SPIN: Hierarchical Segmentation with Subpart Granularity in Natural Images](https://mlsanity.com/paper/2407.09686/) (2024)
- [Iterative Robust Visual Grounding with Masked Reference based Centerpoint Supervision](https://mlsanity.com/paper/2307.12392/) (2023)
- [JPPF: Multi-task Fusion for Consistent Panoptic-Part Segmentation](https://mlsanity.com/paper/2311.18618/) (2023)
- [Mapping the Unseen: Unified Promptable Panoptic Mapping with Dynamic Labeling using Foundation Models](https://mlsanity.com/paper/2405.02162/) (2024)
- [LangHOPS: Language Grounded Hierarchical Open-Vocabulary Part Segmentation](https://mlsanity.com/paper/2510.25263/) (2025)
- [Dictionary-based Framework for Interpretable and Consistent Object Parsing](https://mlsanity.com/paper/2502.19540/) (2025)
- [SurgPIS: Surgical-instrument-level Instances and Part-level Semantics for Weakly-supervised Part-aware Instance Segmentation](https://mlsanity.com/paper/2507.19592/) (2025)

## Abstract (arXiv)

Panoptic Part Segmentation (PPS) unifies panoptic and part segmentation into one task. Previous works utilize separate approaches to handle things, stuff, and part predictions without shared computation and task association. We aim to unify these tasks at the architectural level, designing the first end-to-end unified framework, Panoptic-PartFormer. Moreover, we find the previous metric PartPQ biases to PQ. To handle both issues, we first design a meta-architecture that decouples part features and things/stuff features, respectively. We model things, stuff, and parts as object queries and directly learn to optimize all three forms of prediction as a unified mask prediction and classification problem. We term our model as Panoptic-PartFormer. Second, we propose a new metric Part-Whole Quality (PWQ), better to measure this task from pixel-region and part-whole perspectives. It also decouples the errors for part segmentation and panoptic segmentation. Third, inspired by Mask2Former, based on our meta-architecture, we propose Panoptic-PartFormer++ and design a new part-whole cross-attention scheme to boost part segmentation qualities further. We design a new part-whole interaction method using masked cross attention. Finally, extensive ablation studies and analysis demonstrate the effectiveness of both Panoptic-PartFormer and Panoptic-PartFormer++. Compared with previous Panoptic-PartFormer, our Panoptic-PartFormer++ achieves 2% PartPQ and 3% PWQ improvements on the Cityscapes PPS dataset and 5% PartPQ on the Pascal Context PPS dataset. On both datasets, Panoptic-PartFormer++ achieves new state-of-the-art results. Our models can serve as a strong baseline and aid future research in PPS. The source code and trained models will be available at~urlhttps://github.com/lxtGH/Panoptic-PartFormer.