# Self-supervised Audiovisual Representation Learning for Remote Sensing Data

arXiv: https://arxiv.org/abs/2108.00688 · HTML: https://mlsanity.com/paper/2108.00688/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 64 references that are in the map.
- First posted on arXiv 2 Aug 2021.

## Builds on (most cited first, 30 of 38)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://mlsanity.com/paper/1709.00029/) (2017)
- [Remote Sensing Image Scene Classification: Benchmark and State of the Art](https://mlsanity.com/paper/1703.00121/) (2017)
- [Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey](https://mlsanity.com/paper/1902.06162/) (2019)
- [Albumentations: fast and flexible image augmentations](https://mlsanity.com/paper/1809.06839/) (2018)
- [Look, Listen and Learn](https://mlsanity.com/paper/1705.08168/) (2017)
- [AID: A Benchmark Dataset for Performance Evaluation of Aerial Scene Classification](https://mlsanity.com/paper/1608.05167/) (2016)
- [Audio-Visual Scene Analysis with Self-Supervised Multisensory Features](https://mlsanity.com/paper/1804.03641/) (2018)
- [DeepGlobe 2018: A Challenge to Parse the Earth through Satellite Images](https://mlsanity.com/paper/1805.06561/) (2018)
- [Self-Supervised Learning by Cross-Modal Audio-Video Clustering](https://mlsanity.com/paper/1911.12667/) (2019)
- [The Sound of Pixels](https://mlsanity.com/paper/1804.03160/) (2018)
- [Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization](https://mlsanity.com/paper/1807.00230/) (2018)
- [SoundNet: Learning Sound Representations from Unlabeled Video](https://mlsanity.com/paper/1610.09001/) (2016)
- [Seasonal Contrast: Unsupervised Pre-Training from Uncurated Remote Sensing Data](https://mlsanity.com/paper/2103.16607/) (2021)
- [Geography-Aware Self-Supervised Learning](https://mlsanity.com/paper/2011.09980/) (2020)
- [Learning to Localize Sound Source in Visual Scenes](https://mlsanity.com/paper/1803.03849/) (2018)
- [Visually Indicated Sounds](https://mlsanity.com/paper/1512.08512/) (2015)
- [Deep learning in remote sensing: a review](https://mlsanity.com/paper/1710.03959/) (2017)
- [Multiple Sound Sources Localization from Coarse to Fine](https://mlsanity.com/paper/2007.06355/) (2020)
- [Tile2Vec: Unsupervised representation learning for spatially distributed data](https://mlsanity.com/paper/1805.02855/) (2018)
- [Co-Separating Sounds of Visual Objects](https://mlsanity.com/paper/1904.07750/) (2019)
- [BigEarthNet-MM: A Large Scale Multi-Modal Multi-Label Benchmark Archive for Remote Sensing Image Classification and Retrieval](https://mlsanity.com/paper/2105.07921/) (2021)
- [Self-Supervised Pretraining Improves Self-Supervised Pretraining](https://mlsanity.com/paper/2103.12718/) (2021)
- [Ambient Sound Provides Supervision for Visual Learning](https://mlsanity.com/paper/1608.07017/) (2016)

## Built on by (most cited first, 24 of 24)

- [Self-supervised Learning in Remote Sensing: A Review](https://mlsanity.com/paper/2206.13188/) (2022)
- [Vision-Language Models in Remote Sensing: Current Progress and Future Trends](https://mlsanity.com/paper/2305.05726/) (2023)
- [SSL4EO-S12: A Large-Scale Multi-Modal, Multi-Temporal Dataset for Self-Supervised Learning in Earth Observation](https://mlsanity.com/paper/2211.07044/) (2022)
- [A Billion-scale Foundation Model for Remote Sensing Images](https://mlsanity.com/paper/2304.05215/) (2023)
- [Self-Supervised Multimodal Learning: A Survey](https://mlsanity.com/paper/2304.01008/) (2023)
- [Revisiting pre-trained remote sensing model benchmarks: resizing and normalization matters](https://mlsanity.com/paper/2305.13456/) (2023)
- [Self-supervised remote sensing feature learning: Learning Paradigms, Challenges, and Future Works](https://mlsanity.com/paper/2211.08129/) (2022)
- [Consecutive Pretraining: A Knowledge Transfer Learning Strategy with Relevant Unlabeled Data for Remote Sensing Domain](https://mlsanity.com/paper/2207.03860/) (2022)
- [Multimodal contrastive learning for remote sensing tasks](https://mlsanity.com/paper/2209.02329/) (2022)
- [Learning Tri-modal Embeddings for Zero-Shot Soundscape Mapping](https://mlsanity.com/paper/2309.10667/) (2023)
- [Foundation Models in Remote Sensing: Evolving from Unimodality to Multimodality](https://mlsanity.com/paper/2603.00988/) (2026)
- [Deep Unsupervised Contrastive Hashing for Large-Scale Cross-Modal Text-Image Retrieval in Remote Sensing](https://mlsanity.com/paper/2201.08125/) (2022)
- [GEOBIND: Binding Text, Image, and Audio through Satellite Images](https://mlsanity.com/paper/2404.11720/) (2024)
- [Common Practices and Taxonomy in Deep Multi-view Fusion for Remote Sensing Applications](https://mlsanity.com/paper/2301.01200/) (2022)
- [Supervising Remote Sensing Change Detection Models with 3D Surface Semantics](https://mlsanity.com/paper/2202.13251/) (2022)
- [Self-Supervised In-Domain Representation Learning for Remote Sensing Image Scene Classification](https://mlsanity.com/paper/2302.01793/) (2023)
- [PSM: Learning Probabilistic Embeddings for Multi-scale Zero-Shot Soundscape Mapping](https://mlsanity.com/paper/2408.07050/) (2024)
- [PixelDINO: Semi-Supervised Semantic Segmentation for Detecting Permafrost Disturbances](https://mlsanity.com/paper/2401.09271/) (2024)
- [Multi-modal Co-learning for Earth Observation: Enhancing single-modality models via modality collaboration](https://mlsanity.com/paper/2510.19579/) (2025)
- [Supervised and Contrastive Self-Supervised In-Domain Representation Learning for Dense Prediction Problems in Remote Sensing](https://mlsanity.com/paper/2301.12541/) (2023)
- [HAVE-Net: Hallucinated Audio-Visual Embeddings for Few-Shot Classification with Unimodal Cues](https://mlsanity.com/paper/2309.13470/) (2023)
- [SounDiT: Geo-Contextual Soundscape-to-Landscape Generation](https://mlsanity.com/paper/2505.12734/) (2025)
- [Sat2Sound: A Unified Framework for Zero-Shot Soundscape Mapping](https://mlsanity.com/paper/2505.13777/) (2025)
- [Cross-Modal Urban Sensing: Evaluating Sound-Vision Alignment Across Street-Level and Aerial Imagery](https://mlsanity.com/paper/2506.03388/) (2025)

## Abstract (arXiv)

Many current deep learning approaches make extensive use of backbone networks pre-trained on large datasets like ImageNet, which are then fine-tuned to perform a certain task. In remote sensing, the lack of comparable large annotated datasets and the wide diversity of sensing platforms impedes similar developments. In order to contribute towards the availability of pre-trained backbone networks in remote sensing, we devise a self-supervised approach for pre-training deep neural networks. By exploiting the correspondence between geo-tagged audio recordings and remote sensing imagery, this is done in a completely label-free manner, eliminating the need for laborious manual annotation. For this purpose, we introduce the SoundingEarth dataset, which consists of co-located aerial imagery and audio samples all around the world. Using this dataset, we then pre-train ResNet models to map samples from both modalities into a common embedding space, which encourages the models to understand key properties of a scene that influence both visual and auditory appearance. To validate the usefulness of the proposed approach, we evaluate the transfer learning performance of pre-trained weights obtained against weights obtained through other means. By fine-tuning the models on a number of commonly used remote sensing datasets, we show that our approach outperforms existing pre-training strategies for remote sensing imagery. The dataset, code and pre-trained model weights will be available at https://github.com/khdlr/SoundingEarth.