# SatSwinMAE: Efficient Autoencoding for Multiscale Time-series Satellite Imagery

arXiv: https://arxiv.org/abs/2405.02512 · HTML: https://mlsanity.com/paper/2405.02512/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 23 references that are in the map.
- First posted on arXiv 3 May 2024.

## Builds on (most cited first, 18 of 18)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2105.05537/) (2021)
- [SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery](https://mlsanity.com/paper/2207.08051/) (2022)
- [Scale-MAE: A Scale-Aware Masked Autoencoder for Multiscale Geospatial Representation Learning](https://mlsanity.com/paper/2212.14532/) (2022)
- [When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations](https://mlsanity.com/paper/2106.01548/) (2021)
- [Foundation Models for Generalist Geospatial Artificial Intelligence](https://mlsanity.com/paper/2310.18660/) (2023)
- [Self-supervised Learning in Remote Sensing: A Review](https://mlsanity.com/paper/2206.13188/) (2022)
- [ConvMAE: Masked Convolution Meets Masked Autoencoders](https://mlsanity.com/paper/2205.03892/) (2022)
- [Rethinking Transformers Pre-training for Multi-Spectral Satellite Imagery](https://mlsanity.com/paper/2403.05419/) (2024)
- [PhilEO Bench: Evaluating Geo-Spatial Foundation Models](https://mlsanity.com/paper/2401.04464/) (2024)
- [Representation Learning for Remote Sensing: An Unsupervised Sensor Fusion Approach](https://mlsanity.com/paper/2108.05094/) (2021)
- [Embedding Earth: Self-supervised contrastive pre-training for dense land cover classification](https://mlsanity.com/paper/2203.06041/) (2022)

## Built on by (most cited first, 2 of 2)

- [Large scale cross-regional remote sensing flood monitoring framework for operative mapping and impact analysis](https://mlsanity.com/paper/2607.28401/) (2026)
- [Earth Observation Foundation Models for Terrestrial Ecohydrology: From Representation Learning to Process Inference](https://mlsanity.com/paper/2608.15282/) (2026)

## Abstract (arXiv)

Recent advancements in foundation models have significantly impacted various fields, including natural language processing, computer vision, and multi-modal tasks. One area that stands to benefit greatly is Earth observation, where these models can efficiently process large-scale, unlabeled geospatial data. In this work we extend the SwinMAE model to integrate temporal information for satellite time-series data. The architecture employs a hierarchical 3D Masked Autoencoder (MAE) with Video Swin Transformer blocks to effectively capture multi-scale spatio-temporal dependencies in satellite imagery. To enhance transfer learning, we incorporate both encoder and decoder pretrained weights, along with skip connections to preserve scale-specific information. This forms an architecture similar to SwinUNet with an additional temporal component. Our approach shows significant performance improvements over existing state-of-the-art foundation models for all the evaluated downstream tasks: land cover segmentation, building density prediction, flood mapping, wildfire scar mapping and multi-temporal crop segmentation. Particularly, in the land cover segmentation task of the PhilEO Bench dataset, it outperforms other geospatial foundation models with a 10.4% higher accuracy.