# Using Multiple Input Modalities Can Improve Data-Efficiency and O.O.D. Generalization for ML with Satellite Imagery

arXiv: https://arxiv.org/abs/2507.13385 · HTML: https://mlsanity.com/paper/2507.13385/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 46 references that are in the map.
- First posted on arXiv 15 Jul 2025.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation](https://mlsanity.com/paper/1511.00561/) (2015)
- [Vision Transformers Need Registers](https://mlsanity.com/paper/2309.16588/) (2023)
- [SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery](https://mlsanity.com/paper/2207.08051/) (2022)
- [Scale-MAE: A Scale-Aware Masked Autoencoder for Multiscale Geospatial Representation Learning](https://mlsanity.com/paper/2212.14532/) (2022)
- [BigEarthNet: A Large-Scale Benchmark Archive For Remote Sensing Image Understanding](https://mlsanity.com/paper/1902.06148/) (2019)
- [GeoCLIP: Clip-Inspired Alignment between Locations and Images for Effective Worldwide Geo-localization](https://mlsanity.com/paper/2309.16020/) (2023)
- [SatCLIP: Global, General-Purpose Location Embeddings with Satellite Imagery](https://mlsanity.com/paper/2311.17179/) (2023)
- [Presence-Only Geographical Priors for Fine-Grained Image Classification](https://mlsanity.com/paper/1906.05272/) (2019)
- [A Generalizable and Accessible Approach to Machine Learning with Global Satellite Imagery](https://mlsanity.com/paper/2010.08168/) (2020)
- [MMEarth: Exploring Multi-Modal Pretext Tasks For Geospatial Representation Learning](https://mlsanity.com/paper/2405.02771/) (2024)
- [CSP: Self-Supervised Contrastive Spatial Pre-Training for Geospatial-Visual Representations](https://mlsanity.com/paper/2305.01118/) (2023)
- [SustainBench: Benchmarks for Monitoring the Sustainable Development Goals with Machine Learning](https://mlsanity.com/paper/2111.04724/) (2021)
- [Geo-Aware Networks for Fine-Grained Recognition](https://mlsanity.com/paper/1906.01737/) (2019)
- [Urban2Vec: Incorporating Street View Imagery and POIs for Multi-Modal Urban Neighborhood Embedding](https://mlsanity.com/paper/2001.11101/) (2020)
- [Improving Image Classification with Location Context](https://mlsanity.com/paper/1505.03873/) (2015)
- [Revisiting the Power of Prompt for Visual Tuning](https://mlsanity.com/paper/2402.02382/) (2024)
- [Better, Not Just More: Data-Centric Machine Learning for Earth Observation](https://mlsanity.com/paper/2312.05327/) (2023)
- [Multi-modal learning for geospatial vegetation forecasting](https://mlsanity.com/paper/2303.16198/) (2023)
- [Farmland Parcel Delineation Using Spatio-temporal Convolutional Networks](https://mlsanity.com/paper/2004.05471/) (2020)

## Built on by (most cited first, 5 of 5)

- [MMEarth-Bench: Global Model Adaptation via Multimodal Test-Time Training](https://mlsanity.com/paper/2602.06285/) (2026)
- [A Proxy Consistency Loss for Grounded Fusion of Earth Observation and Location Encoders](https://mlsanity.com/paper/2604.18881/) (2026)
- [Better Together: Evaluating the Complementarity of Earth Embedding Models](https://mlsanity.com/paper/2605.18667/) (2026)
- [Hedgementation = Hedgerow Segmentation: A Remote Sensing Benchmark](https://mlsanity.com/paper/2606.23615/) (2026)
- [Planetary Feature Fields are Scalable Earth Representations](https://mlsanity.com/paper/2609.37784/) (2026)

## Abstract (arXiv)

A large variety of geospatial data layers is available around the world ranging from remotely-sensed raster data like satellite imagery, digital elevation models, predicted land cover maps, and human-annotated data, to data derived from environmental sensors such as air temperature or wind speed data. A large majority of machine learning models trained on satellite imagery (SatML), however, are designed primarily for optical input modalities such as multi-spectral satellite imagery. To better understand the value of using other input modalities alongside optical imagery in supervised learning settings, we generate augmented versions of SatML benchmark tasks by appending additional geographic data layers to datasets spanning classification, regression, and segmentation. Using these augmented datasets, we find that fusing additional geographic inputs with optical imagery can significantly improve SatML model performance. Benefits are largest in settings where labeled data are limited and in geographic out-of-sample settings, suggesting that multi-modal inputs may be especially valuable for data-efficiency and out-of-sample performance of SatML models. Surprisingly, we find that hard-coded fusion strategies outperform learned variants, with interesting implications for future work.