# Panoptic Segmentation Meets Remote Sensing

arXiv: https://arxiv.org/abs/2111.12126 · HTML: https://mlsanity.com/paper/2111.12126/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 93 references that are in the map.
- First posted on arXiv 23 Nov 2021.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018)
- [DOTA: A Large-scale Dataset for Object Detection in Aerial Images](https://mlsanity.com/paper/1711.10398/) (2017)
- [Res2Net: A New Multi-scale Backbone Architecture](https://mlsanity.com/paper/1904.01169/) (2019)
- [YOLACT: Real-time Instance Segmentation](https://mlsanity.com/paper/1904.02689/) (2019)
- [Mask Scoring R-CNN](https://mlsanity.com/paper/1903.00241/) (2019)
- [iSAID: A Large-scale Dataset for Instance Segmentation in Aerial Images](https://mlsanity.com/paper/1905.12886/) (2019)
- [SpaceNet: A Remote Sensing Dataset and Challenge Series](https://mlsanity.com/paper/1807.01232/) (2018)
- [Panoptic Segmentation of Satellite Image Time Series with Convolutional Temporal Attention Networks](https://mlsanity.com/paper/2107.07933/) (2021)
- [EfficientPS: Efficient Panoptic Segmentation](https://mlsanity.com/paper/2004.02307/) (2020)
- [YOLACT++: Better Real-time Instance Segmentation](https://mlsanity.com/paper/1912.06218/) (2019)
- [Fully Convolutional Adaptation Networks for Semantic Segmentation](https://mlsanity.com/paper/1804.08286/) (2018)
- [Towards Better Exploiting Convolutional Neural Networks for Remote Sensing Scene Classification](https://mlsanity.com/paper/1602.01517/) (2016)
- [Boundary Loss for Remote Sensing Imagery Semantic Segmentation](https://mlsanity.com/paper/1905.07852/) (2019)
- [Vehicle Instance Segmentation from Aerial Image and Video Using a Multi-Task Learning Residual Fully Convolutional Network](https://mlsanity.com/paper/1805.10485/) (2018)
- [Linear Span Network for Object Skeleton Detection](https://mlsanity.com/paper/1807.09601/) (2018)

## Built on by (most cited first, 14 of 14)

- [The Segment Anything Model (SAM) for Remote Sensing Applications: From Zero to One Shot](https://mlsanity.com/paper/2306.16623/) (2023)
- [A Sentinel-2 multi-year, multi-country benchmark dataset for crop classification and segmentation with deep learning](https://mlsanity.com/paper/2204.00951/) (2022)
- [Deep Learning for Satellite Image Time Series Analysis: A Review](https://mlsanity.com/paper/2404.03936/) (2024)
- [Dense Out-of-Distribution Detection by Robust Learning on Synthetic Negative Data](https://mlsanity.com/paper/2112.12833/) (2021)
- [Panoptic Perception: A Novel Task and Fine-grained Dataset for Universal Remote Sensing Image Interpretation](https://mlsanity.com/paper/2404.04608/) (2024)
- [PRUE: A Practical Recipe for Field Boundary Segmentation at Scale](https://mlsanity.com/paper/2603.27101/) (2026)
- [Panoptic SwiftNet: Pyramidal Fusion for Real-time Panoptic Segmentation](https://mlsanity.com/paper/2203.07908/) (2022)
- [DEARLi: Decoupled Enhancement of Recognition and Localization for Semi-supervised Panoptic Segmentation](https://mlsanity.com/paper/2507.10118/) (2025)
- [Towards On-Board Panoptic Segmentation of Multispectral Satellite Images](https://mlsanity.com/paper/2204.01952/) (2022)
- [Agricultural Landscape Understanding At Country-Scale](https://mlsanity.com/paper/2411.05359/) (2024)
- [Remote SAMsing: From Segment Anything to Segment Everything](https://mlsanity.com/paper/2605.00256/) (2026)
- [Offshore Wind Plant Instance Segmentation Using Sentinel-1 Time Series, GIS, and Semantic Segmentation Models](https://mlsanity.com/paper/2312.08773/) (2023)
- [Deep Learning Meets OBIA: Tasks, Challenges, Strategies, and Perspectives](https://mlsanity.com/paper/2408.01607/) (2024)
- [iSAGE: A Human-in-the-Loop Framework for Remote Sensing Semantic Segmentation via Sparse Point Supervision](https://mlsanity.com/paper/2606.10136/) (2026)

## Abstract (arXiv)

Panoptic segmentation combines instance and semantic predictions, allowing the detection of "things" and "stuff" simultaneously. Effectively approaching panoptic segmentation in remotely sensed data can be auspicious in many challenging problems since it allows continuous mapping and specific target counting. Several difficulties have prevented the growth of this task in remote sensing: (a) most algorithms are designed for traditional images, (b) image labelling must encompass "things" and "stuff" classes, and (c) the annotation format is complex. Thus, aiming to solve and increase the operability of panoptic segmentation in remote sensing, this study has five objectives: (1) create a novel data preparation pipeline for panoptic segmentation, (2) propose an annotation conversion software to generate panoptic annotations; (3) propose a novel dataset on urban areas, (4) modify the Detectron2 for the task, and (5) evaluate difficulties of this task in the urban setting. We used an aerial image with a 0,24-meter spatial resolution considering 14 classes. Our pipeline considers three image inputs, and the proposed software uses point shapefiles for creating samples in the COCO format. Our study generated 3,400 samples with 512x512 pixel dimensions. We used the Panoptic-FPN with two backbones (ResNet-50 and ResNet-101), and the model evaluation considered semantic instance and panoptic metrics. We obtained 93.9, 47.7, and 64.9 for the mean IoU, box AP, and PQ. Our study presents the first effective pipeline for panoptic segmentation and an extensive database for other researchers to use and deal with other data or related problems requiring a thorough scene understanding.