# Exploring DINO: Emergent Properties and Limitations for Synthetic Aperture Radar Imagery

arXiv: https://arxiv.org/abs/2310.03513 · HTML: https://mlsanity.com/paper/2310.03513/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 14 references that are in the map.
- First posted on arXiv 5 Oct 2023.

## Builds on (most cited first, 9 of 9)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Geography-Aware Self-Supervised Learning](https://mlsanity.com/paper/2011.09980/) (2020)
- [Self-supervised Learning in Remote Sensing: A Review](https://mlsanity.com/paper/2206.13188/) (2022)
- [Deep Learning Meets SAR](https://mlsanity.com/paper/2006.10027/) (2020)
- [Self-Supervised Learning of Remote Sensing Scene Representations Using Contrastive Multiview Coding](https://mlsanity.com/paper/2104.07070/) (2021)

## Built on by (most cited first, 2 of 2)

- [PANGAEA: A Global and Inclusive Benchmark for Geospatial Foundation Models](https://mlsanity.com/paper/2412.04204/) (2024)
- [M3LEO: A Multi-Modal, Multi-Label Earth Observation Dataset Integrating Interferometric SAR and Multispectral Data](https://mlsanity.com/paper/2406.04230/) (2024)

## Abstract (arXiv)

Self-supervised learning (SSL) models have recently demonstrated remarkable performance across various tasks, including image segmentation. This study delves into the emergent characteristics of the Self-Distillation with No Labels (DINO) algorithm and its application to Synthetic Aperture Radar (SAR) imagery. We pre-train a vision transformer (ViT)-based DINO model using unlabeled SAR data, and later fine-tune the model to predict high-resolution land cover maps. We rigorously evaluate the utility of attention maps generated by the ViT backbone and compare them with the model's token embedding space. We observe a small improvement in model performance with pre-training compared to training from scratch and discuss the limitations and opportunities of SSL for remote sensing and land cover segmentation. Beyond small performance increases, we show that ViT attention maps hold great intrinsic value for remote sensing, and could provide useful inputs to other algorithms. With this, our work lays the groundwork for bigger and better SSL models for Earth Observation.