# CSMoE: An Efficient Remote Sensing Foundation Model with Soft Mixture-of-Experts

arXiv: https://arxiv.org/abs/2509.14104 · HTML: https://mlsanity.com/paper/2509.14104/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Vision mixture-of-experts, one of 12 directions in Mixture-of-experts.
- Builds on 33 of its 47 references that are in the map.
- First posted on arXiv 17 Sep 2025.

## Placed in

- Mixture-of-experts / [Vision mixture-of-experts](https://mlsanity.com/moe/vision-mixture-of-experts/)

## Builds on (most cited first, 30 of 33)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [Learning Representations by Maximizing Mutual Information Across Views](https://mlsanity.com/paper/1906.00910/) (2019)
- [SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery](https://mlsanity.com/paper/2207.08051/) (2022)
- [Scale-MAE: A Scale-Aware Masked Autoencoder for Multiscale Geospatial Representation Learning](https://mlsanity.com/paper/2212.14532/) (2022)
- [From Sparse to Soft Mixtures of Experts](https://mlsanity.com/paper/2308.00951/) (2023)
- [UNetFormer: A UNet-like Transformer for Efficient Semantic Segmentation of Remote Sensing Urban Scene Imagery](https://mlsanity.com/paper/2109.08937/) (2021)
- [SatlasPretrain: A Large-Scale Dataset for Remote Sensing Image Understanding](https://mlsanity.com/paper/2211.15660/) (2022)
- [Prithvi-EO-2.0: A Versatile Multi-Temporal Foundation Model for Earth Observation Applications](https://mlsanity.com/paper/2412.02732/) (2024)
- [GEO-Bench: Toward Foundation Models for Earth Monitoring](https://mlsanity.com/paper/2306.03831/) (2023)
- [Neural Plasticity-Inspired Multimodal Foundation Model for Earth Observation](https://mlsanity.com/paper/2403.15356/) (2024)
- [TerraMind: Large-Scale Generative Multimodality for Earth Observation](https://mlsanity.com/paper/2504.11171/) (2025)
- [BigEarthNet-MM: A Large Scale Multi-Modal Multi-Label Benchmark Archive for Remote Sensing Image Classification and Retrieval](https://mlsanity.com/paper/2105.07921/) (2021)
- [Remote Sensing Image Scene Classification Meets Deep Learning: Challenges, Methods, Benchmarks, and Opportunities](https://mlsanity.com/paper/2005.01094/) (2020)
- [Foundation Models for Remote Sensing and Earth Observation: A Survey](https://mlsanity.com/paper/2410.16602/) (2024)
- [Vision Foundation Models in Remote Sensing: A Survey](https://mlsanity.com/paper/2408.03464/) (2024)
- [Automatic Data Curation for Self-Supervised Learning: A Clustering-Based Approach](https://mlsanity.com/paper/2405.15613/) (2024)
- [reBEN: Refined BigEarthNet Dataset for Remote Sensing Image Analysis](https://mlsanity.com/paper/2407.03653/) (2024)
- [Major TOM: Expandable Datasets for Earth Observation](https://mlsanity.com/paper/2402.12095/) (2024)
- [RSUniVLM: A Unified Vision Language Model for Remote Sensing via Granularity-oriented Mixture of Experts](https://mlsanity.com/paper/2412.05679/) (2024)
- [TerraTorch: The Geospatial Foundation Models Toolkit](https://mlsanity.com/paper/2503.20563/) (2025)
- [Better, Not Just More: Data-Centric Machine Learning for Earth Observation](https://mlsanity.com/paper/2312.05327/) (2023)
- [RingMoE: Mixture-of-Modality-Experts Multi-Modal Foundation Models for Universal Remote Sensing Image Interpretation](https://mlsanity.com/paper/2504.03166/) (2025)
- [RS-MoE: A Vision-Language Model with Mixture of Experts for Remote Sensing Image Captioning and Visual Question Answering](https://mlsanity.com/paper/2411.01595/) (2024)
- [Exploring Masked Autoencoders for Sensor-Agnostic Image Retrieval in Remote Sensing](https://mlsanity.com/paper/2401.07782/) (2024)
- [OReole-FM: successes and challenges toward billion-parameter foundation models for high-resolution satellite imagery](https://mlsanity.com/paper/2410.19965/) (2024)
- [A Novel Self-Supervised Cross-Modal Image Retrieval Method In Remote Sensing](https://mlsanity.com/paper/2202.11429/) (2022)

## Built on by (most cited first, 3 of 3)

- [How Much of a Model Do We Need? Redundancy and Slimmability in Remote Sensing Foundation Models](https://mlsanity.com/paper/2601.22841/) (2026)
- [CR-JEPA: Cross-Modal Joint-Embedding Predictive Learning for Remote Sensing Image Retrieval](https://mlsanity.com/paper/2606.00706/) (2026)
- [MEOX: Compact Multimodal Mixture-of-Experts for Earth Observation](https://mlsanity.com/paper/2609.05351/) (2026)

## Abstract (arXiv)

Self-supervised learning (SSL) through masked autoencoders (MAEs) has recently attracted great attention for remote sensing (RS) foundation model (FM) development, enabling improved representation learning across diverse sensors and downstream tasks. However, existing RS FMs often either suffer from substantial computational complexity during both training and inference or exhibit limited representational capacity. In addition, their pretraining datasets can contain redundant images, which increases training cost without improving the learned representations. These issues restrict their practical applicability in RS. To address these limitations, we improve the computational efficiency of RS FMs along two axes: i) model efficiency; and ii) training-data efficiency. Model efficiency is achieved by integrating the Soft mixture-of-experts (MoE) mechanism into the FM, which allows modality-specific expert processing alongside shared cross-sensor representation learning while reducing computational complexity at both training and inference time. We apply this adaptation to the Cross-Sensor Masked Autoencoder (CSMAE) model, which serves as our main baseline, resulting in the Cross-Sensor Mixture-of-Experts (CSMoE) model. Training-data efficiency is achieved by a thematic-climatic descriptor-driven sampling strategy, which constructs a reduced training set from a large-scale image archive while retaining its geographic and thematic-climatic diversity, and thus reduces pretraining cost. Extensive experiments on scene classification, semantic segmentation, and content-based image retrieval (CBIR) show that CSMoE remains competitive with state-of-the-art RS FMs while requiring substantially fewer floating-point operations (FLOPs). The associated code for the model and the training set creation, as well as the pretrained model weights, will be available at https://git.tu-berlin.de/rsim/csmoe.