# Fully Convolutional Networks for Panoptic Segmentation with Point-based Supervision

arXiv: https://arxiv.org/abs/2108.07682 · HTML: https://mlsanity.com/paper/2108.07682/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 76 of its 89 references that are in the map.
- First posted on arXiv 17 Aug 2021.

## Builds on (most cited first, 30 of 76)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Objects as Points](https://mlsanity.com/paper/1904.07850/) (2019)
- [Dual Attention Network for Scene Segmentation](https://mlsanity.com/paper/1809.02983/) (2018)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018)
- [Path Aggregation Network for Instance Segmentation](https://mlsanity.com/paper/1803.01534/) (2018)
- [CornerNet: Detecting Objects as Paired Keypoints](https://mlsanity.com/paper/1808.01244/) (2018)
- [CCNet: Criss-Cross Attention for Semantic Segmentation](https://mlsanity.com/paper/1811.11721/) (2018)
- [Deformable ConvNets v2: More Deformable, Better Results](https://mlsanity.com/paper/1811.11168/) (2018)
- [Panoptic Feature Pyramid Networks](https://mlsanity.com/paper/1901.02446/) (2019)
- [Segmentation Transformer: Object-Contextual Representations for Semantic Segmentation](https://mlsanity.com/paper/1909.11065/) (2019)
- [YOLACT: Real-time Instance Segmentation](https://mlsanity.com/paper/1904.02689/) (2019)
- [Panoptic-DeepLab](https://mlsanity.com/paper/1910.04751/) (2019)
- [MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers](https://mlsanity.com/paper/2012.00759/) (2020)
- [ScribbleSup: Scribble-Supervised Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1604.05144/) (2016)

## Built on by (most cited first, 30 of 31)

- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Panoptic SegFormer: Delving Deeper into Panoptic Segmentation with Transformers](https://mlsanity.com/paper/2109.03814/) (2021)
- [High-Quality Entity Segmentation](https://mlsanity.com/paper/2211.05776/) (2022)
- [Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labeling](https://mlsanity.com/paper/2111.12698/) (2021)
- [Mapping Degeneration Meets Label Evolution: Learning Infrared Small Target Detection with Single Point Supervision](https://mlsanity.com/paper/2304.01484/) (2023)
- [Panoptic-PartFormer: Learning a Unified Model for Panoptic Part Segmentation](https://mlsanity.com/paper/2204.04655/) (2022)
- [Unsupervised Universal Image Segmentation](https://mlsanity.com/paper/2312.17243/) (2023)
- [CoMFormer: Continual Learning in Semantic and Panoptic Segmentation](https://mlsanity.com/paper/2211.13999/) (2022)
- [Multi-Scale Aligned Distillation for Low-Resolution Detection](https://mlsanity.com/paper/2109.06875/) (2021)
- [Point2Mask: Point-supervised Panoptic Segmentation via Optimal Transport](https://mlsanity.com/paper/2308.01779/) (2023)
- [Pointly-Supervised Panoptic Segmentation](https://mlsanity.com/paper/2210.13950/) (2022)
- [Monte Carlo Linear Clustering with Single-Point Supervision is Enough for Infrared Small Target Detection](https://mlsanity.com/paper/2304.04442/) (2023)
- [Mean Shift Mask Transformer for Unseen Object Instance Segmentation](https://mlsanity.com/paper/2211.11679/) (2022)
- [PanopticPartFormer++: A Unified and Decoupled View for Panoptic Part Segmentation](https://mlsanity.com/paper/2301.00954/) (2023)
- [Pair then Relation: Pair-Net for Panoptic Scene Graph Generation](https://mlsanity.com/paper/2307.08699/) (2023)
- [Active Pointly-Supervised Instance Segmentation](https://mlsanity.com/paper/2207.11493/) (2022)
- [High-Quality Entity Segmentation and Grounding](https://mlsanity.com/paper/2402.02555/) (2024)
- [STRAP: Structured Object Affordance Segmentation with Point Supervision](https://mlsanity.com/paper/2304.08492/) (2023)
- [CA-SSL: Class-Agnostic Semi-Supervised Learning for Detection and Segmentation](https://mlsanity.com/paper/2112.04966/) (2021)
- [DejaVu: Conditional Regenerative Learning to Enhance Dense Prediction](https://mlsanity.com/paper/2303.01573/) (2023)
- [3D Medical Image Segmentation with Sparse Annotation via Cross-Teaching between 3D and 2D Networks](https://mlsanity.com/paper/2307.16256/) (2023)
- [Label-efficient Segmentation via Affinity Propagation](https://mlsanity.com/paper/2310.10533/) (2023)
- [A Good Foundation is Worth Many Labels: Label-Efficient Panoptic Segmentation](https://mlsanity.com/paper/2405.19035/) (2024)
- [What is Point Supervision Worth in Video Instance Segmentation?](https://mlsanity.com/paper/2404.01990/) (2024)
- [End-to-End Instance Edge Detection](https://mlsanity.com/paper/2204.02898/) (2022)
- [Advancing Visual Large Language Model for Multi-granular Versatile Perception](https://mlsanity.com/paper/2507.16213/) (2025)
- [Enhancing Weakly Supervised 3D Medical Image Segmentation through Probabilistic-aware Learning](https://mlsanity.com/paper/2403.02566/) (2024)
- [SSTD: Stripe-Like Space Target Detection Using Single-Point Weak Supervision](https://mlsanity.com/paper/2407.18097/) (2024)
- [Seg-VAR: Image Segmentation with Visual Autoregressive Modeling](https://mlsanity.com/paper/2511.12594/) (2025)
- [Click on Mask: A Labor-efficient Annotation Framework with Level Set for Infrared Small Target Detection](https://mlsanity.com/paper/2310.12562/) (2023)

## Abstract (arXiv)

In this paper, we present a conceptually simple, strong, and efficient framework for fully- and weakly-supervised panoptic segmentation, called Panoptic FCN. Our approach aims to represent and predict foreground things and background stuff in a unified fully convolutional pipeline, which can be optimized with point-based fully or weak supervision. In particular, Panoptic FCN encodes each object instance or stuff category with the proposed kernel generator and produces the prediction by convolving the high-resolution feature directly. With this approach, instance-aware and semantically consistent properties for things and stuff can be respectively satisfied in a simple generate-kernel-then-segment workflow. Without extra boxes for localization or instance separation, the proposed approach outperforms the previous box-based and -free models with high efficiency. Furthermore, we propose a new form of point-based annotation for weakly-supervised panoptic segmentation. It only needs several random points for both things and stuff, which dramatically reduces the annotation cost of human. The proposed Panoptic FCN is also proved to have much superior performance in this weakly-supervised setting, which achieves 82% of the fully-supervised performance with only 20 randomly annotated points per instance. Extensive experiments demonstrate the effectiveness and efficiency of Panoptic FCN on COCO, VOC 2012, Cityscapes, and Mapillary Vistas datasets. And it sets up a new leading benchmark for both fully- and weakly-supervised panoptic segmentation. Our code and models are made publicly available at https://github.com/dvlab-research/PanopticFCN.