# Panoptic Segmentation of Satellite Image Time Series with Convolutional Temporal Attention Networks

arXiv: https://arxiv.org/abs/2107.07933 · HTML: https://mlsanity.com/paper/2107.07933/

## Key facts

- Cited by 143 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 56 references that are in the map.
- First posted on arXiv 16 Jul 2021.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [Objects as Points](https://mlsanity.com/paper/1904.07850/) (2019)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [CornerNet: Detecting Objects as Paired Keypoints](https://mlsanity.com/paper/1808.01244/) (2018)
- [Panoptic Feature Pyramid Networks](https://mlsanity.com/paper/1901.02446/) (2019)
- [Deep Layer Aggregation](https://mlsanity.com/paper/1707.06484/) (2017)
- [Video Instance Segmentation](https://mlsanity.com/paper/1905.04804/) (2019)
- [Deep learning in agriculture: A survey](https://mlsanity.com/paper/1807.11809/) (2018)
- [Delving Deeper into Convolutional Networks for Learning Video Representations](https://mlsanity.com/paper/1511.06432/) (2015)
- [Video Panoptic Segmentation](https://mlsanity.com/paper/2006.11339/) (2020)
- [EfficientPS: Efficient Panoptic Segmentation](https://mlsanity.com/paper/2004.02307/) (2020)
- [Temporal Convolutional Neural Network for the Classification of Satellite Image Time Series](https://mlsanity.com/paper/1811.10166/) (2018)
- [Self-attention for raw optical Satellite Time Series Classification](https://mlsanity.com/paper/1910.10536/) (2019)
- [Satellite Image Time Series Classification with Pixel-Set Encoders and Temporal Self-Attention](https://mlsanity.com/paper/1911.07757/) (2019)
- [Lightweight Temporal Self-Attention for Classifying Satellite Image Time Series](https://mlsanity.com/paper/2007.00586/) (2020)
- [Crop mapping from image time series: deep learning with multi-scale label hierarchies](https://mlsanity.com/paper/2102.08820/) (2021)
- [Learning to Segment Moving Objects](https://mlsanity.com/paper/1712.01127/) (2017)
- [Deep learning on edge: extracting field boundaries from satellite images with a convolutional neural network](https://mlsanity.com/paper/1910.12023/) (2019)
- [CenterMask: single shot instance segmentation with point representation](https://mlsanity.com/paper/2004.04446/) (2020)
- [Semantic Segmentation of Earth Observation Data Using Multimodal and Multi-scale Deep Networks](https://mlsanity.com/paper/1609.06846/) (2016)
- [Land Cover Classification via Multi-temporal Spatial Data by Recurrent Neural Networks](https://mlsanity.com/paper/1704.04055/) (2017)
- [Time-Space tradeoff in deep learning models for crop classification on satellite multi-spectral image time series](https://mlsanity.com/paper/1901.10503/) (2019)
- [Crop Classification under Varying Cloud Cover with Neural Ordinary Differential Equations](https://mlsanity.com/paper/2012.02542/) (2020)

## Built on by (most cited first, 30 of 143)

- [SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery](https://mlsanity.com/paper/2207.08051/) (2022)
- [SatlasPretrain: A Large-Scale Dataset for Remote Sensing Image Understanding](https://mlsanity.com/paper/2211.15660/) (2022)
- [Prithvi-EO-2.0: A Versatile Multi-Temporal Foundation Model for Earth Observation Applications](https://mlsanity.com/paper/2412.02732/) (2024)
- [DynamicEarthNet: Daily Multi-Spectral Satellite Dataset for Semantic Change Segmentation](https://mlsanity.com/paper/2203.12560/) (2022)
- [AnySat: One Earth Observation Model for Many Resolutions, Scales, and Modalities](https://mlsanity.com/paper/2412.14123/) (2024)
- [TorchGeo: Deep Learning With Geospatial Data](https://mlsanity.com/paper/2111.08872/) (2021)
- [PANGAEA: A Global and Inclusive Benchmark for Geospatial Foundation Models](https://mlsanity.com/paper/2412.04204/) (2024)
- [OmniSat: Self-Supervised Modality Fusion for Earth Observation](https://mlsanity.com/paper/2404.08351/) (2024)
- [ViTs for SITS: Vision Transformers for Satellite Image Time Series](https://mlsanity.com/paper/2301.04944/) (2023)
- [GEOBench-VLM: Benchmarking Vision-Language Models for Geospatial Tasks](https://mlsanity.com/paper/2411.19325/) (2024)
- [Multi-Modal Temporal Attention Models for Crop Mapping from Satellite Time Series](https://mlsanity.com/paper/2112.07558/) (2021)
- [Galileo: Learning Global & Local Features of Many Remote Sensing Modalities](https://mlsanity.com/paper/2502.09356/) (2025)
- [FLAIR: a Country-Scale Land Cover Semantic Segmentation Dataset From Multi-Source Optical Imagery](https://mlsanity.com/paper/2310.13336/) (2023)
- [EarthNets: Empowering AI in Earth Observation](https://mlsanity.com/paper/2210.04936/) (2022)
- [MMST-ViT: Climate Change-aware Crop Yield Prediction via Multi-Modal Spatial-Temporal Vision Transformer](https://mlsanity.com/paper/2309.09067/) (2023)
- [UnCRtainTS: Uncertainty Quantification for Cloud Removal in Optical Satellite Time Series](https://mlsanity.com/paper/2304.05464/) (2023)
- [What a MESS: Multi-Domain Evaluation of Zero-Shot Semantic Segmentation](https://mlsanity.com/paper/2306.15521/) (2023)
- [Fields of The World: A Machine Learning Benchmark Dataset For Global Agricultural Field Boundary Segmentation](https://mlsanity.com/paper/2409.16252/) (2024)
- [OlmoEarth: Stable Latent Image Modeling for Multimodal Earth Observation](https://mlsanity.com/paper/2511.13655/) (2025)
- [Self-Supervised Learning for Invariant Representations from Multi-Spectral and SAR Images](https://mlsanity.com/paper/2205.02049/) (2022)
- [A Sentinel-2 multi-year, multi-country benchmark dataset for crop classification and segmentation with deep learning](https://mlsanity.com/paper/2204.00951/) (2022)
- [Better, Not Just More: Data-Centric Machine Learning for Earth Observation](https://mlsanity.com/paper/2312.05327/) (2023)
- [Vision-Language Modeling Meets Remote Sensing: Models, Datasets and Perspectives](https://mlsanity.com/paper/2505.14361/) (2025)
- [OpenForest: A data catalogue for machine learning in forest monitoring](https://mlsanity.com/paper/2311.00277/) (2023)
- [Panoptic Segmentation Meets Remote Sensing](https://mlsanity.com/paper/2111.12126/) (2021)
- [Generalized Classification of Satellite Image Time Series with Thermal Positional Encoding](https://mlsanity.com/paper/2203.09175/) (2022)
- [TerraMesh: A Planetary Mosaic of Multimodal Earth Observation Data](https://mlsanity.com/paper/2504.11172/) (2025)
- [Deep Learning for Satellite Image Time Series Analysis: A Review](https://mlsanity.com/paper/2404.03936/) (2024)
- [Effective Cloud Removal for Remote Sensing Images by an Improved Mean-Reverting Denoising Model with Elucidated Design Space](https://mlsanity.com/paper/2503.23717/) (2025)
- [FUSU: A Multi-temporal-source Land Use Change Segmentation Dataset for Fine-grained Urban Semantic Understanding](https://mlsanity.com/paper/2405.19055/) (2024)

## Abstract (arXiv)

Unprecedented access to multi-temporal satellite imagery has opened new perspectives for a variety of Earth observation tasks. Among them, pixel-precise panoptic segmentation of agricultural parcels has major economic and environmental implications. While researchers have explored this problem for single images, we argue that the complex temporal patterns of crop phenology are better addressed with temporal sequences of images. In this paper, we present the first end-to-end, single-stage method for panoptic segmentation of Satellite Image Time Series (SITS). This module can be combined with our novel image sequence encoding network which relies on temporal self-attention to extract rich and adaptive multi-scale spatio-temporal features. We also introduce PASTIS, the first open-access SITS dataset with panoptic annotations. We demonstrate the superiority of our encoder for semantic segmentation against multiple competing architectures, and set up the first state-of-the-art of panoptic segmentation of SITS. Our implementation and PASTIS are publicly available.