# S4: Self-Supervised Sensing Across the Spectrum

arXiv: https://arxiv.org/abs/2405.01656 · HTML: https://mlsanity.com/paper/2405.01656/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 61 references that are in the map.
- First posted on arXiv 2 May 2024.

## Builds on (most cited first, 30 of 31)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation](https://mlsanity.com/paper/1606.06650/) (2016)
- [Image Inpainting for Irregular Holes Using Partial Convolutions](https://mlsanity.com/paper/1804.07723/) (2018)
- [SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery](https://mlsanity.com/paper/2207.08051/) (2022)
- [Contrastive learning of global and local features for medical image segmentation with limited annotations](https://mlsanity.com/paper/2006.10511/) (2020)
- [Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/2011.10043/) (2020)
- [Seasonal Contrast: Unsupervised Pre-Training from Uncurated Remote Sensing Data](https://mlsanity.com/paper/2103.16607/) (2021)
- [Geography-Aware Self-Supervised Learning](https://mlsanity.com/paper/2011.09980/) (2020)
- [PseudoSeg: Designing Pseudo Labels for Semantic Segmentation](https://mlsanity.com/paper/2010.09713/) (2020)
- [Panoptic Segmentation of Satellite Image Time Series with Convolutional Temporal Attention Networks](https://mlsanity.com/paper/2107.07933/) (2021)
- [Self-Supervision with Superpixels: Training Few-shot Medical Image Segmentation without Annotation](https://mlsanity.com/paper/2007.09886/) (2020)
- [Pixel Contrastive-Consistent Semi-Supervised Semantic Segmentation](https://mlsanity.com/paper/2108.09025/) (2021)
- [Bootstrapping Semantic Segmentation with Regional Contrast](https://mlsanity.com/paper/2104.04465/) (2021)
- [DynamicEarthNet: Daily Multi-Spectral Satellite Dataset for Semantic Change Segmentation](https://mlsanity.com/paper/2203.12560/) (2022)
- [Satellite Image Time Series Classification with Pixel-Set Encoders and Temporal Self-Attention](https://mlsanity.com/paper/1911.07757/) (2019)
- [Multi-Modal Temporal Attention Models for Crop Mapping from Satellite Time Series](https://mlsanity.com/paper/2112.07558/) (2021)
- [Self-supervised Contrastive Representation Learning for Semi-supervised Time-Series Classification](https://mlsanity.com/paper/2208.06616/) (2022)
- [Multi-Sensor Data Fusion for Cloud Removal in Global and All-Season Sentinel-2 Imagery](https://mlsanity.com/paper/2009.07683/) (2020)
- [Leverage Your Local and Global Representations: A New Self-Supervised Learning Strategy](https://mlsanity.com/paper/2203.17205/) (2022)
- [Multi3Net: Segmenting Flooded Buildings via Fusion of Multiresolution, Multisensor, and Multitemporal Satellite Imagery](https://mlsanity.com/paper/1812.01756/) (2018)
- [Spatial-Temporal Super-Resolution of Satellite Imagery via Conditional Pixel Synthesis](https://mlsanity.com/paper/2106.11485/) (2021)
- [Contrastive Learning of Image Representations with Cross-Video Cycle-Consistency](https://mlsanity.com/paper/2105.06463/) (2021)
- [Seeing Through Clouds in Satellite Images](https://mlsanity.com/paper/2106.08408/) (2021)
- [Fully Self-Supervised Learning for Semantic Segmentation](https://mlsanity.com/paper/2202.11981/) (2022)
- [Segmenting across places: The need for fair transfer learning with satellite imagery](https://mlsanity.com/paper/2204.04358/) (2022)
- [Monitoring War Destruction from Space: A Machine Learning Approach](https://mlsanity.com/paper/2010.05970/) (2020)
- [Self-supervised Contrastive Learning for Irrigation Detection in Satellite Imagery](https://mlsanity.com/paper/2108.05484/) (2021)

## Built on by (most cited first, 5 of 5)

- [LEO-Split: A Semi-Supervised Split Learning Framework over LEO Satellite Networks](https://mlsanity.com/paper/2501.01293/) (2025)
- [A Satellite-Ground Synergistic Large Vision-Language Model System for Earth Observation](https://mlsanity.com/paper/2507.05731/) (2025)
- [Towards Scalable Foundation Model for Multi-modal and Hyperspectral Geospatial Data](https://mlsanity.com/paper/2503.12843/) (2025)
- [SeeFar: Satellite Agnostic Multi-Resolution Dataset for Geospatial Foundation Models](https://mlsanity.com/paper/2406.06776/) (2024)
- [RS-OOD: A Vision-Language Augmented Framework for Out-of-Distribution Detection in Remote Sensing](https://mlsanity.com/paper/2509.02273/) (2025)

## Abstract (arXiv)

Satellite image time series (SITS) segmentation is crucial for many applications like environmental monitoring, land cover mapping and agricultural crop type classification. However, training models for SITS segmentation remains a challenging task due to the lack of abundant training data, which requires fine grained annotation. We propose S4 a new self-supervised pre-training approach that significantly reduces the requirement for labeled training data by utilizing two new insights: (a) Satellites capture images in different parts of the spectrum such as radio frequencies, and visible frequencies. (b) Satellite imagery is geo-registered allowing for fine-grained spatial alignment. We use these insights to formulate pre-training tasks in S4. We also curate m2s2-SITS, a large-scale dataset of unlabeled, spatially-aligned, multi-modal and geographic specific SITS that serves as representative pre-training data for S4. Finally, we evaluate S4 on multiple SITS segmentation datasets and demonstrate its efficacy against competing baselines while using limited labeled data.