# Auto-Panoptic: Cooperative Multi-Component Architecture Search for Panoptic Segmentation

arXiv: https://arxiv.org/abs/2010.16119 · HTML: https://mlsanity.com/paper/2010.16119/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Oct 2020.

## Built on by (most cited first, 12 of 12)

- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [Panoptic SegFormer: Delving Deeper into Panoptic Segmentation with Transformers](https://mlsanity.com/paper/2109.03814/) (2021)
- [Video K-Net: A Simple, Strong, and Unified Baseline for Video Segmentation](https://mlsanity.com/paper/2204.04656/) (2022)
- [Panoptic-PartFormer: Learning a Unified Model for Panoptic Part Segmentation](https://mlsanity.com/paper/2204.04655/) (2022)
- [PolyphonicFormer: Unified Query Learning for Depth-aware Video Panoptic Segmentation](https://mlsanity.com/paper/2112.02582/) (2021)
- [Pointly-Supervised Panoptic Segmentation](https://mlsanity.com/paper/2210.13950/) (2022)
- [PanopticPartFormer++: A Unified and Decoupled View for Panoptic Part Segmentation](https://mlsanity.com/paper/2301.00954/) (2023)
- [Panoptic Segmentation: A Review](https://mlsanity.com/paper/2111.10250/) (2021)
- [Pair then Relation: Pair-Net for Panoptic Scene Graph Generation](https://mlsanity.com/paper/2307.08699/) (2023)
- [Neural Architecture Search for Dense Prediction Tasks in Computer Vision](https://mlsanity.com/paper/2202.07242/) (2022)
- [Panoptic, Instance and Semantic Relations: A Relational Context Encoder to Enhance Panoptic Segmentation](https://mlsanity.com/paper/2204.05370/) (2022)
- [NASOA: Towards Faster Task-oriented Online Fine-tuning with a Zoo of Models](https://mlsanity.com/paper/2108.03434/) (2021)

## Abstract (arXiv)

Panoptic segmentation is posed as a new popular test-bed for the state-of-the-art holistic scene understanding methods with the requirement of simultaneously segmenting both foreground things and background stuff. The state-of-the-art panoptic segmentation network exhibits high structural complexity in different network components, i.e. backbone, proposal-based foreground branch, segmentation-based background branch, and feature fusion module across branches, which heavily relies on expert knowledge and tedious trials. In this work, we propose an efficient, cooperative and highly automated framework to simultaneously search for all main components including backbone, segmentation branches, and feature fusion module in a unified panoptic segmentation pipeline based on the prevailing one-shot Network Architecture Search (NAS) paradigm. Notably, we extend the common single-task NAS into the multi-component scenario by taking the advantage of the newly proposed intra-modular search space and problem-oriented inter-modular search space, which helps us to obtain an optimal network architecture that not only performs well in both instance segmentation and semantic segmentation tasks but also be aware of the reciprocal relations between foreground things and background stuff classes. To relieve the vast computation burden incurred by applying NAS to complicated network architectures, we present a novel path-priority greedy search policy to find a robust, transferrable architecture with significantly reduced searching overhead. Our searched architecture, namely Auto-Panoptic, achieves the new state-of-the-art on the challenging COCO and ADE20K benchmarks. Moreover, extensive experiments are conducted to demonstrate the effectiveness of path-priority policy and transferability of Auto-Panoptic across different datasets. Codes and models are available at: https://github.com/Jacobew/AutoPanoptic.