# Towards DeepSentinel: An extensible corpus of labelled Sentinel-1 and -2 imagery and a general-purpose sensor-fusion semantic embedding model

arXiv: https://arxiv.org/abs/2102.06260 · HTML: https://mlsanity.com/paper/2102.06260/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 38 references that are in the map.
- First posted on arXiv 11 Feb 2021.

## Builds on (most cited first, 10 of 10)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Self-Attention Generative Adversarial Networks](https://mlsanity.com/paper/1805.08318/) (2018)
- [Tile2Vec: Unsupervised representation learning for spatially distributed data](https://mlsanity.com/paper/1805.02855/) (2018)
- [Multi-temporal Sentinel-1 and -2 Data Fusion for Optical Image Simulation](https://mlsanity.com/paper/1807.09954/) (2018)

## Built on by (most cited first, 2 of 2)

- [CROMA: Remote Sensing Representations with Contrastive Radar-Optical Masked Autoencoders](https://mlsanity.com/paper/2311.00566/) (2023)
- [Transfer Learning with Pretrained Remote Sensing Transformers](https://mlsanity.com/paper/2209.14969/) (2022)

## Abstract (arXiv)

Earth observation offers new insight into anthropogenic changes to nature, and how these changes are effecting (and are effected by) the built environment and the real economy. With the global availability of medium-resolution (10-30m) synthetic aperture radar (SAR) Sentinel-1 and multispectral Sentinel-2 imagery, machine learning can be employed to offer these insights at scale, unbiased to the reporting of companies and countries. In this paper, I introduce DeepSentinel, a data pipeline and experimentation framework for producing general-purpose semantic embeddings of paired Sentinel-1 and Sentinel-2 imagery. I document the development of an extensible corpus of labelled and unlabelled imagery for the purposes of sensor fusion research. With this new dataset I develop a set of experiments applying popular self-supervision methods and encoder architectures to a land cover classification problem. Tile2vec spatial encoding with a self-attention enabled ResNet model outperforms deeper ResNet variants as well as pretraining with variational autoencoding and contrastive loss. All supporting and derived data and code are made publicly available.