# Context-self contrastive pretraining for crop type semantic segmentation

arXiv: https://arxiv.org/abs/2104.04310 · HTML: https://mlsanity.com/paper/2104.04310/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 73 references that are in the map.
- First posted on arXiv 9 Apr 2021.

## Builds on (most cited first, 30 of 42)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation](https://mlsanity.com/paper/1606.06650/) (2016)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Contrastive Learning with Hard Negative Samples](https://mlsanity.com/paper/2010.04592/) (2020)
- [Stand-Alone Self-Attention in Vision Models](https://mlsanity.com/paper/1906.05909/) (2019)
- [Dense Contrastive Learning for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2011.09157/) (2020)
- [Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge](https://mlsanity.com/paper/1811.02629/) (2018)
- [PointRend: Image Segmentation as Rendering](https://mlsanity.com/paper/1912.08193/) (2019)
- [Hard Negative Mixing for Contrastive Learning](https://mlsanity.com/paper/2010.01028/) (2020)
- [Attention Augmented Convolutional Networks](https://mlsanity.com/paper/1904.09925/) (2019)
- [3D MRI brain tumor segmentation using autoencoder regularization](https://mlsanity.com/paper/1810.11654/) (2018)
- [A Metric Learning Reality Check](https://mlsanity.com/paper/2003.08505/) (2020)
- [Lip Reading Sentences in the Wild](https://mlsanity.com/paper/1611.05358/) (2016)
- [Deep learning in agriculture: A survey](https://mlsanity.com/paper/1807.11809/) (2018)
- [Unsupervised Learning of Dense Visual Representations](https://mlsanity.com/paper/2011.05499/) (2020)

## Built on by (most cited first, 11 of 11)

- [ViTs for SITS: Vision Transformers for Satellite Image Time Series](https://mlsanity.com/paper/2301.04944/) (2023)
- [Deep Learning for Satellite Image Time Series Analysis: A Review](https://mlsanity.com/paper/2404.03936/) (2024)
- [Semi-Supervised Semantic Segmentation with Cross Teacher Training](https://mlsanity.com/paper/2209.01327/) (2022)
- [Embedding Earth: Self-supervised contrastive pre-training for dense land cover classification](https://mlsanity.com/paper/2203.06041/) (2022)
- [Exact: Exploring Space-Time Perceptive Clues for Weakly Supervised Satellite Image Time Series Semantic Segmentation](https://mlsanity.com/paper/2412.03968/) (2024)
- [DeepSatData: Building large scale datasets of satellite images for training machine learning models](https://mlsanity.com/paper/2104.13824/) (2021)
- [Detecting Looted Archaeological Sites from Satellite Image Time Series](https://mlsanity.com/paper/2409.09432/) (2024)
- [VistaFormer: Scalable Vision Transformers for Satellite Image Time Series Segmentation](https://mlsanity.com/paper/2409.08461/) (2024)
- [Low-Resource Crop Classification from Multi-Spectral Time Series Using Lossless Compressors](https://mlsanity.com/paper/2405.18119/) (2024)
- [Semantic-Aware Representation of Multi-Modal Data for Data Ingress: A Literature Review](https://mlsanity.com/paper/2407.12438/) (2024)
- [TEA: Temporal Adaptive Satellite Image Semantic Segmentation](https://mlsanity.com/paper/2601.04956/) (2026)

## Abstract (arXiv)

In this paper, we propose a fully supervised pre-training scheme based on contrastive learning particularly tailored to dense classification tasks. The proposed Context-Self Contrastive Loss (CSCL) learns an embedding space that makes semantic boundaries pop-up by use of a similarity metric between every location in a training sample and its local context. For crop type semantic segmentation from Satellite Image Time Series (SITS) we find performance at parcel boundaries to be a critical bottleneck and explain how CSCL tackles the underlying cause of that problem, improving the state-of-the-art performance in this task. Additionally, using images from the Sentinel-2 (S2) satellite missions we compile the largest, to our knowledge, SITS dataset densely annotated by crop type and parcel identities, which we make publicly available together with the data generation pipeline. Using that data we find CSCL, even with minimal pre-training, to improve all respective baselines and present a process for semantic segmentation at super-resolution for obtaining crop classes at a more granular level. The code and instructions to download the data can be found in https://github.com/michaeltrs/DeepSatModels.