# Exploring Masked Autoencoders for Sensor-Agnostic Image Retrieval in Remote Sensing

arXiv: https://arxiv.org/abs/2401.07782 · HTML: https://mlsanity.com/paper/2401.07782/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 48 references that are in the map.
- First posted on arXiv 15 Jan 2024.

## Builds on (most cited first, 22 of 22)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [Learning Representations by Maximizing Mutual Information Across Views](https://mlsanity.com/paper/1906.00910/) (2019)
- [Exploring Plain Vision Transformer Backbones for Object Detection](https://mlsanity.com/paper/2203.16527/) (2022)
- [ImageNet-21K Pretraining for the Masses](https://mlsanity.com/paper/2104.10972/) (2021)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery](https://mlsanity.com/paper/2207.08051/) (2022)
- [Scale-MAE: A Scale-Aware Masked Autoencoder for Multiscale Geospatial Representation Learning](https://mlsanity.com/paper/2212.14532/) (2022)
- [Advancing Plain Vision Transformer Towards Remote Sensing Foundation Model](https://mlsanity.com/paper/2208.03987/) (2022)
- [BigEarthNet-MM: A Large Scale Multi-Modal Multi-Label Benchmark Archive for Remote Sensing Image Classification and Retrieval](https://mlsanity.com/paper/2105.07921/) (2021)
- [Masked Vision and Language Modeling for Multi-modal Representation Learning](https://mlsanity.com/paper/2208.02131/) (2022)
- [A Billion-scale Foundation Model for Remote Sensing Images](https://mlsanity.com/paper/2304.05215/) (2023)
- [CMID: A Unified Self-Supervised Learning Framework for Remote Sensing Image Understanding](https://mlsanity.com/paper/2304.09670/) (2023)
- [USat: A Unified Self-Supervised Encoder for Multi-Sensor Satellite Imagery](https://mlsanity.com/paper/2312.02199/) (2023)
- [Deep Learning for Image Search and Retrieval in Large Remote Sensing Archives](https://mlsanity.com/paper/2004.01613/) (2020)
- [A Novel Self-Supervised Cross-Modal Image Retrieval Method In Remote Sensing](https://mlsanity.com/paper/2202.11429/) (2022)
- [CMIR-NET : A Deep Learning Based Model For Cross-Modal Retrieval In Remote Sensing](https://mlsanity.com/paper/1904.04794/) (2019)
- [A Novel Graph-Theoretic Deep Representation Learning Method for Multi-Label Remote Sensing Image Retrieval](https://mlsanity.com/paper/2106.00506/) (2021)
- [Generative Reasoning Integrated Label Noise Robust Deep Image Representation Learning](https://mlsanity.com/paper/2212.01261/) (2022)

## Built on by (most cited first, 12 of 12)

- [Mission Critical -- Satellite Data is a Distinct Modality in Machine Learning](https://mlsanity.com/paper/2402.01444/) (2024)
- [PANGAEA: A Global and Inclusive Benchmark for Geospatial Foundation Models](https://mlsanity.com/paper/2412.04204/) (2024)
- [OmniSat: Self-Supervised Modality Fusion for Earth Observation](https://mlsanity.com/paper/2404.08351/) (2024)
- [Foundation Models in Remote Sensing: Evolving from Unimodality to Multimodality](https://mlsanity.com/paper/2603.00988/) (2026)
- [Increasing the Robustness of Model Predictions to Missing Sensors in Earth Observation](https://mlsanity.com/paper/2407.15512/) (2024)
- [REJEPA: A Novel Joint-Embedding Predictive Architecture for Efficient Remote Sensing Image Retrieval](https://mlsanity.com/paper/2504.03169/) (2025)
- [CSMoE: An Efficient Remote Sensing Foundation Model with Soft Mixture-of-Experts](https://mlsanity.com/paper/2509.14104/) (2025)
- [Multispectral to Hyperspectral using Pretrained Foundational model](https://mlsanity.com/paper/2502.19451/) (2025)
- [Continual Self-Supervised Learning with Masked Autoencoders in Remote Sensing](https://mlsanity.com/paper/2506.21312/) (2025)
- [Rank-based Geographical Regularization: Revisiting Contrastive Self-Supervised Learning for Multispectral Remote Sensing Imagery](https://mlsanity.com/paper/2601.02289/) (2026)
- [Rethinking Electro-Optical Vision Foundation Models for Remote Sensing Retrieval: A Controlled Comparison with Generalist VFM](https://mlsanity.com/paper/2605.02283/) (2026)
- [CR-JEPA: Cross-Modal Joint-Embedding Predictive Learning for Remote Sensing Image Retrieval](https://mlsanity.com/paper/2606.00706/) (2026)

## Abstract (arXiv)

Self-supervised learning through masked autoencoders (MAEs) has recently attracted great attention for remote sensing (RS) image representation learning, and thus embodies a significant potential for content-based image retrieval (CBIR) from ever-growing RS image archives. However, the existing MAE based CBIR studies in RS assume that the considered RS images are acquired by a single image sensor, and thus are only suitable for uni-modal CBIR problems. The effectiveness of MAEs for cross-sensor CBIR, which aims to search semantically similar images across different image modalities, has not been explored yet. In this paper, we take the first step to explore the effectiveness of MAEs for sensor-agnostic CBIR in RS. To this end, we present a systematic overview on the possible adaptations of the vanilla MAE to exploit masked image modeling on multi-sensor RS image archives (denoted as cross-sensor masked autoencoders [CSMAEs]) in the context of CBIR. Based on different adjustments applied to the vanilla MAE, we introduce different CSMAE models. We also provide an extensive experimental analysis of these CSMAE models. We finally derive a guideline to exploit masked image modeling for uni-modal and cross-modal CBIR problems in RS. The code of this work is publicly available at https://github.com/jakhac/CSMAE.