# Revisiting the Encoding of Satellite Image Time Series

arXiv: https://arxiv.org/abs/2305.02086 · HTML: https://mlsanity.com/paper/2305.02086/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 46 references that are in the map.
- First posted on arXiv 3 May 2023.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Are Transformers Effective for Time Series Forecasting?](https://mlsanity.com/paper/2205.13504/) (2022)
- [FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting](https://mlsanity.com/paper/2201.12740/) (2022)
- [Attention-based Deep Multiple Instance Learning](https://mlsanity.com/paper/1802.04712/) (2018)
- [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018)
- [PVT v2: Improved Baselines with Pyramid Vision Transformer](https://mlsanity.com/paper/2106.13797/) (2021)
- [GroupViT: Semantic Segmentation Emerges from Text Supervision](https://mlsanity.com/paper/2202.11094/) (2022)
- [Rethinking Positional Encoding in Language Pre-training](https://mlsanity.com/paper/2006.15595/) (2020)
- [FlexiViT: One Model for All Patch Sizes](https://mlsanity.com/paper/2212.08013/) (2022)
- [ETSformer: Exponential Smoothing Transformers for Time-series Forecasting](https://mlsanity.com/paper/2202.01381/) (2022)
- [Panoptic Segmentation of Satellite Image Time Series with Convolutional Temporal Attention Networks](https://mlsanity.com/paper/2107.07933/) (2021)
- [Set Functions for Time Series](https://mlsanity.com/paper/1909.12064/) (2019)
- [Tile2Vec: Unsupervised representation learning for spatially distributed data](https://mlsanity.com/paper/1805.02855/) (2018)
- [Interpolation-Prediction Networks for Irregularly Sampled Time Series](https://mlsanity.com/paper/1909.07782/) (2019)
- [Temporal Convolutional Neural Network for the Classification of Satellite Image Time Series](https://mlsanity.com/paper/1811.10166/) (2018)
- [Satellite Image Time Series Classification with Pixel-Set Encoders and Temporal Self-Attention](https://mlsanity.com/paper/1911.07757/) (2019)
- [Multi-Temporal Land Cover Classification with Sequential Recurrent Encoders](https://mlsanity.com/paper/1802.02080/) (2018)
- [CMT-DeepLab: Clustering Mask Transformers for Panoptic Segmentation](https://mlsanity.com/paper/2206.08948/) (2022)
- [ViTs for SITS: Vision Transformers for Satellite Image Time Series](https://mlsanity.com/paper/2301.04944/) (2023)
- [Lightweight Temporal Self-Attention for Classifying Satellite Image Time Series](https://mlsanity.com/paper/2007.00586/) (2020)
- [Image as Set of Points](https://mlsanity.com/paper/2303.01494/) (2023)
- [Revisiting the Transferability of Supervised Pretraining: an MLP Perspective](https://mlsanity.com/paper/2112.00496/) (2021)

## Built on by (most cited first, 6 of 6)

- [Deep Learning for Satellite Image Time Series Analysis: A Review](https://mlsanity.com/paper/2404.03936/) (2024)
- [Paving the way toward foundation models for irregular and unaligned Satellite Image Time Series](https://mlsanity.com/paper/2407.08448/) (2024)
- [Tree semantic segmentation from aerial image time series](https://mlsanity.com/paper/2407.13102/) (2024)
- [VistaFormer: Scalable Vision Transformers for Satellite Image Time Series Segmentation](https://mlsanity.com/paper/2409.08461/) (2024)
- [A Joint Learning Framework with Feature Reconstruction and Prediction for Incomplete Satellite Image Time Series in Agricultural Semantic Segmentation](https://mlsanity.com/paper/2505.19159/) (2025)
- [The Last Remote Sensing Model Built by Humans? Casting Any Earth Observation Task as Recursive Self-Improvement](https://mlsanity.com/paper/2610.18845/) (2026)

## Abstract (arXiv)

Satellite Image Time Series (SITS) representation learning is complex due to high spatiotemporal resolutions, irregular acquisition times, and intricate spatiotemporal interactions. These challenges result in specialized neural network architectures tailored for SITS analysis. The field has witnessed promising results achieved by pioneering researchers, but transferring the latest advances or established paradigms from Computer Vision (CV) to SITS is still highly challenging due to the existing suboptimal representation learning framework. In this paper, we develop a novel perspective of SITS processing as a direct set prediction problem, inspired by the recent trend in adopting query-based transformer decoders to streamline the object detection or image segmentation pipeline. We further propose to decompose the representation learning process of SITS into three explicit steps: collect-update-distribute, which is computationally efficient and suits for irregularly-sampled and asynchronous temporal satellite observations. Facilitated by the unique reformulation, our proposed temporal learning backbone of SITS, initially pre-trained on the resource efficient pixel-set format and then fine-tuned on the downstream dense prediction tasks, has attained new state-of-the-art (SOTA) results on the PASTIS benchmark dataset. Specifically, the clear separation between temporal and spatial components in the semantic/panoptic segmentation pipeline of SITS makes us leverage the latest advances in CV, such as the universal image segmentation architecture, resulting in a noticeable 2.5 points increase in mIoU and 8.8 points increase in PQ, respectively, compared to the best scores reported so far.