# Mission Critical -- Satellite Data is a Distinct Modality in Machine Learning

arXiv: https://arxiv.org/abs/2402.01444 · HTML: https://mlsanity.com/paper/2402.01444/

## Key facts

- Cited by 77 papers in the map (data through 14 Oct 2026).
- Builds on 62 of its 117 references that are in the map.
- First posted on arXiv 2 Feb 2024.

## Builds on (most cited first, 30 of 62)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [Measuring Robustness to Natural Distribution Shifts in Image Classification](https://mlsanity.com/paper/2007.00644/) (2020)
- [SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery](https://mlsanity.com/paper/2207.08051/) (2022)
- [Functional Map of the World](https://mlsanity.com/paper/1711.07846/) (2017)
- [Scale-MAE: A Scale-Aware Masked Autoencoder for Multiscale Geospatial Representation Learning](https://mlsanity.com/paper/2212.14532/) (2022)
- [Tackling Climate Change with Machine Learning](https://mlsanity.com/paper/1906.05433/) (2019)
- [BigEarthNet: A Large-Scale Benchmark Archive For Remote Sensing Image Understanding](https://mlsanity.com/paper/1902.06148/) (2019)
- [Seasonal Contrast: Unsupervised Pre-Training from Uncurated Remote Sensing Data](https://mlsanity.com/paper/2103.16607/) (2021)
- [Geography-Aware Self-Supervised Learning](https://mlsanity.com/paper/2011.09980/) (2020)
- [GeoCLIP: Clip-Inspired Alignment between Locations and Images for Effective Worldwide Geo-localization](https://mlsanity.com/paper/2309.16020/) (2023)
- [CROMA: Remote Sensing Representations with Contrastive Radar-Optical Masked Autoencoders](https://mlsanity.com/paper/2311.00566/) (2023)
- [SatlasPretrain: A Large-Scale Dataset for Remote Sensing Image Understanding](https://mlsanity.com/paper/2211.15660/) (2022)
- [SatCLIP: Global, General-Purpose Location Embeddings with Satellite Imagery](https://mlsanity.com/paper/2311.17179/) (2023)
- [Deep learning in remote sensing: a review](https://mlsanity.com/paper/1710.03959/) (2017)
- [GEO-Bench: Toward Foundation Models for Earth Monitoring](https://mlsanity.com/paper/2306.03831/) (2023)
- [Predicting with Confidence on Unseen Distributions](https://mlsanity.com/paper/2107.03315/) (2021)
- [Tile2Vec: Unsupervised representation learning for spatially distributed data](https://mlsanity.com/paper/1805.02855/) (2018)
- [Presence-Only Geographical Priors for Fine-Grained Image Classification](https://mlsanity.com/paper/1906.05272/) (2019)
- [An Empirical Study of Remote Sensing Pretraining](https://mlsanity.com/paper/2204.02825/) (2022)
- [Are Labels Always Necessary for Classifier Accuracy Evaluation?](https://mlsanity.com/paper/2007.02915/) (2020)
- [SEN12MS -- A Curated Dataset of Georeferenced Multi-Spectral Sentinel-1/2 Imagery for Deep Learning and Data Fusion](https://mlsanity.com/paper/1906.07789/) (2019)
- [Agreement-on-the-Line: Predicting the Performance of Neural Networks under Distribution Shift](https://mlsanity.com/paper/2206.13089/) (2022)
- [Predicting Ground-Level Scene Layout from Aerial Imagery](https://mlsanity.com/paper/1612.02709/) (2016)
- [Multi-Scale Representation Learning for Spatial Feature Distributions using Grid Cells](https://mlsanity.com/paper/2003.00824/) (2020)

## Built on by (most cited first, 30 of 77)

- [PANGAEA: A Global and Inclusive Benchmark for Geospatial Foundation Models](https://mlsanity.com/paper/2412.04204/) (2024)
- [OmniSat: Self-Supervised Modality Fusion for Earth Observation](https://mlsanity.com/paper/2404.08351/) (2024)
- [SegEarth-R1: Geospatial Pixel Reasoning via Large Language Model](https://mlsanity.com/paper/2504.09644/) (2025)
- [Panopticon: Advancing Any-Sensor Foundation Models for Earth Observation](https://mlsanity.com/paper/2503.10845/) (2025)
- [On the Foundations of Earth and Climate Foundation Models](https://mlsanity.com/paper/2405.04285/) (2024)
- [GeoSynth: Contextually-Aware High-Resolution Satellite Image Synthesis](https://mlsanity.com/paper/2404.06637/) (2024)
- [SegEarth-OV: Towards Training-Free Open-Vocabulary Segmentation for Remote Sensing Images](https://mlsanity.com/paper/2410.01768/) (2024)
- [Specialized Foundation Models Struggle to Beat Supervised Baselines](https://mlsanity.com/paper/2411.02796/) (2024)
- [Mind the Modality Gap: Towards a Remote Sensing Vision-Language Model via Cross-modal Alignment](https://mlsanity.com/paper/2402.09816/) (2024)
- [LISAT: Language-Instructed Segmentation Assistant for Satellite Imagery](https://mlsanity.com/paper/2505.02829/) (2025)
- [Foundation Models in Remote Sensing: Evolving from Unimodality to Multimodality](https://mlsanity.com/paper/2603.00988/) (2026)
- [No One Knows the State of the Art in Geospatial Foundation Models](https://mlsanity.com/paper/2605.12678/) (2026)
- [EarthShift: a benchmark for measuring robustness to real-world distribution shifts in Earth observation](https://mlsanity.com/paper/2605.29330/) (2026)
- [Distribution Shifts at Scale: Out-of-distribution Detection in Earth Observation](https://mlsanity.com/paper/2412.13394/) (2024)
- [Geospatial Foundation Models to Enable Progress on Sustainable Development Goals](https://mlsanity.com/paper/2505.24528/) (2025)
- [SegEarth-R2: Towards Comprehensive Language-guided Segmentation for Remote Sensing Images](https://mlsanity.com/paper/2512.20013/) (2025)
- [Annotation-Free Open-Vocabulary Segmentation for Remote-Sensing Images](https://mlsanity.com/paper/2508.18067/) (2025)
- [THOR: A Versatile Foundation Model for Earth Observation Climate and Society Applications](https://mlsanity.com/paper/2601.16011/) (2026)
- [Contrasting local and global modeling with machine learning and satellite data: A case study estimating tree canopy height in African savannas](https://mlsanity.com/paper/2411.14354/) (2024)
- [Be the Change You Want to See: Revisiting Remote Sensing Change Detection Practices](https://mlsanity.com/paper/2507.03367/) (2025)
- [How Does the Spatial Distribution of Pre-training Data Affect Geospatial Foundation Models?](https://mlsanity.com/paper/2501.12535/) (2025)
- [DescribeEarth: Describe Anything for Remote Sensing Images](https://mlsanity.com/paper/2509.25654/) (2025)
- [Responsible AI for Earth Observation](https://mlsanity.com/paper/2405.20868/) (2024)
- [Thicker and Quicker: A Jumbo Token for Fast Plain Vision Transformers](https://mlsanity.com/paper/2502.15021/) (2025)
- [A Genealogy of Foundation Models in Remote Sensing](https://mlsanity.com/paper/2504.17177/) (2025)
- [GeoCrossBench: Cross-Band Generalization for Remote Sensing](https://mlsanity.com/paper/2511.02831/) (2025)
- [KidSat: satellite imagery to map childhood poverty dataset and benchmark](https://mlsanity.com/paper/2407.05986/) (2024)
- [Prototype-Based Methods in Explainable AI and Emerging Opportunities in the Geosciences](https://mlsanity.com/paper/2410.19856/) (2024)
- [On the Effectiveness of Methods and Metrics for Explainable AI in Remote Sensing Image Scene Classification](https://mlsanity.com/paper/2507.05916/) (2025)
- [ConInfer: Context-Aware Inference for Training-Free Open-Vocabulary Remote Sensing Segmentation](https://mlsanity.com/paper/2603.29271/) (2026)

## Abstract (arXiv)

Satellite data has the potential to inspire a seismic shift for machine learning -- one in which we rethink existing practices designed for traditional data modalities. As machine learning for satellite data (SatML) gains traction for its real-world impact, our field is at a crossroads. We can either continue applying ill-suited approaches, or we can initiate a new research agenda that centers around the unique characteristics and challenges of satellite data. This position paper argues that satellite data constitutes a distinct modality for machine learning research and that we must recognize it as such to advance the quality and impact of SatML research across theory, methods, and deployment. We outline critical discussion questions and actionable suggestions to transform SatML from merely an intriguing application area to a dedicated research discipline that helps move the needle on big challenges for machine learning and society.