# Enhancing Self-Supervised Learning for Remote Sensing with Elevation Data: A Case Study with Scarce And High Level Semantic Labels

arXiv: https://arxiv.org/abs/2304.06857 · HTML: https://mlsanity.com/paper/2304.06857/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 16 references that are in the map.
- First posted on arXiv 13 Apr 2023.

## Builds on (most cited first, 10 of 10)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Tile2Vec: Unsupervised representation learning for spatially distributed data](https://mlsanity.com/paper/1805.02855/) (2018)
- [Self-Supervised Learning of Remote Sensing Scene Representations Using Contrastive Multiview Coding](https://mlsanity.com/paper/2104.07070/) (2021)
- [Learning from Label Proportions with Consistency Regularization](https://mlsanity.com/paper/1910.13188/) (2019)
- [The color out of space: learning self-supervised representations for Earth Observation imagery](https://mlsanity.com/paper/2006.12119/) (2020)
- [Global and Local Contrastive Self-Supervised Learning for Semantic Segmentation of HR Remote Sensing Images](https://mlsanity.com/paper/2106.10605/) (2021)
- [Representation Learning for Remote Sensing: An Unsupervised Sensor Fusion Approach](https://mlsanity.com/paper/2108.05094/) (2021)
- [Learning from Label Proportions: A Mutual Contamination Framework](https://mlsanity.com/paper/2006.07330/) (2020)

## Abstract (arXiv)

This work proposes a hybrid unsupervised and supervised learning method to pre-train models applied in Earth observation downstream tasks when only a handful of labels denoting very general semantic concepts are available. We combine a contrastive approach to pre-train models with a pixel-wise regression pre-text task to predict coarse elevation maps, which are commonly available worldwide. We hypothesize that this will allow the model to pre-learn useful representations, as there is generally some correlation between elevation maps and targets in many remote sensing tasks. We assess the performance of our approach on a binary semantic segmentation task and a binary image classification task, both derived from a dataset created for the northwest of Colombia. In both cases, we pre-train our models with 39k unlabeled images, fine-tune them on the downstream tasks with only 80 labeled images, and evaluate them with 2944 labeled images. Our experiments show that our methods, GLCNet+Elevation for segmentation, and SimCLR+Elevation for classification, outperform their counterparts without the pixel-wise regression pre-text task, namely SimCLR and GLCNet, in terms of macro-average F1 Score and Mean Intersection over Union (MIoU). Our study not only encourages the development of pre-training methods that leverage readily available geographical information, such as elevation data, to enhance the performance of self-supervised methods when applied to Earth observation tasks, but also promotes the use of datasets with high-level semantic labels, which are more likely to be updated frequently. Project code can be found in this link hrefhttps://github.com/omarcastano/Elevation-Aware-SSLhttps://github.com/omarcastano/Elevation-Aware-SSL.