# SS-MAE: Spatial-Spectral Masked Auto-Encoder for Multi-Source Remote Sensing Image Classification

arXiv: https://arxiv.org/abs/2311.04442 · HTML: https://mlsanity.com/paper/2311.04442/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 55 references that are in the map.
- First posted on arXiv 8 Nov 2023.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](https://mlsanity.com/paper/2202.03555/) (2022)
- [SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery](https://mlsanity.com/paper/2207.08051/) (2022)
- [How Do Vision Transformers Work?](https://mlsanity.com/paper/2202.06709/) (2022)
- [OmniMAE: Single Model Masked Pretraining on Images and Videos](https://mlsanity.com/paper/2206.08356/) (2022)
- [Multimodal Fusion Transformer for Remote Sensing Image Classification](https://mlsanity.com/paper/2203.16952/) (2022)
- [Multimodal Remote Sensing Benchmark Datasets for Land Cover Classification with A Shared and Specific Feature Learning Model](https://mlsanity.com/paper/2105.10196/) (2021)
- [CSformer: Bridging Convolution and Transformer for Compressive Sensing](https://mlsanity.com/paper/2112.15299/) (2021)
- [Nearest Neighbor-Based Contrastive Learning for Hyperspectral and LiDAR Data Classification](https://mlsanity.com/paper/2301.03335/) (2023)
- [A3CLNN: Spatial, Spectral and Multiscale Attention ConvLSTM Neural Network for Multisource Remote Sensing Data Classification](https://mlsanity.com/paper/2204.04462/) (2022)
- [Knowledge Amalgamation for Object Detection with Transformers](https://mlsanity.com/paper/2203.03187/) (2022)
- [Riemannian Nearest-Regularized Subspace Classification for Polarimetric SAR images](https://mlsanity.com/paper/2201.00337/) (2022)

## Built on by (most cited first, 24 of 24)

- [HyperSIGMA: Hyperspectral Intelligence Comprehension Foundation Model](https://mlsanity.com/paper/2406.11519/) (2024)
- [Depth Any Canopy: Leveraging Depth Foundation Models for Canopy Height Estimation](https://mlsanity.com/paper/2408.04523/) (2024)
- [Geospatial Foundation Models to Enable Progress on Sustainable Development Goals](https://mlsanity.com/paper/2505.24528/) (2025)
- [Hybrid Convolutional and Attention Network for Hyperspectral Image Denoising](https://mlsanity.com/paper/2403.10067/) (2024)
- [Adaptive Frequency Enhancement Network for Remote Sensing Image Semantic Segmentation](https://mlsanity.com/paper/2504.02647/) (2025)
- [ChA-MAEViT: Unifying Channel-Aware Masked Autoencoders and Multi-Channel Vision Transformers for Improved Cross-Channel Learning](https://mlsanity.com/paper/2503.19331/) (2025)
- [Dynamic Cross-Modal Feature Interaction Network for Hyperspectral and LiDAR Data Classification](https://mlsanity.com/paper/2503.06945/) (2025)
- [Towards Scalable Foundation Model for Multi-modal and Hyperspectral Geospatial Data](https://mlsanity.com/paper/2503.12843/) (2025)
- [CARL: Camera-Agnostic Representation Learning for Spectral Image Analysis](https://mlsanity.com/paper/2504.19223/) (2025)
- [MAESTRO: Masked AutoEncoders for Multimodal, Multitemporal, and Multispectral Earth Observation Data](https://mlsanity.com/paper/2508.10894/) (2025)
- [SpecTM: Spectral Targeted Masking for Trustworthy Foundation Models](https://mlsanity.com/paper/2603.22097/) (2026)
- [AI-Driven HSI: Multimodality, Fusion, Challenges, and the Deep Learning Revolution](https://mlsanity.com/paper/2502.06894/) (2025)
- [Language-Informed Hyperspectral Image Synthesis for Imbalanced-Small Sample Classification via Semi-Supervised Conditional Diffusion Model](https://mlsanity.com/paper/2502.19700/) (2025)
- [HyperspectralMAE: The Hyperspectral Imagery Classification Model using Fourier-Encoded Dual-Branch Masked Autoencoder](https://mlsanity.com/paper/2505.05710/) (2025)
- [HyperPointFormer: Multimodal Fusion in 3D Space with Dual-Branch Cross-Attention Transformers](https://mlsanity.com/paper/2505.23206/) (2025)
- [Balanced Diffusion-Guided Fusion for Multimodal Remote Sensing Classification](https://mlsanity.com/paper/2509.23310/) (2025)
- [SpecAware: A Spectral-Content Aware Foundation Model for Unifying Multi-Sensor Learning in Hyperspectral Remote Sensing Mapping](https://mlsanity.com/paper/2510.27219/) (2025)
- [Learning with less: label-efficient land cover classification at very high spatial resolution using self-supervised deep learning](https://mlsanity.com/paper/2511.03004/) (2025)
- [SAMBA: A Scatter-Guided Masked Bidirectional Mamba Foundation Model for SAR Target Recognition](https://mlsanity.com/paper/2606.31668/) (2026)
- [Boosting Multimodal Remote Sensing Image Classification with Transformer-based Heterogeneously Salient Graph Representation](https://mlsanity.com/paper/2311.10320/) (2023)
- [Exploring Generalizable Pre-training for Real-world Change Detection via Geometric Estimation](https://mlsanity.com/paper/2504.14306/) (2025)
- [Prototype-Based Information Compensation Network for Multi-Source Remote Sensing Data Classification](https://mlsanity.com/paper/2505.04003/) (2025)
- [SPAR: Self-supervised Placement-Aware Representation Learning for Distributed Sensing](https://mlsanity.com/paper/2505.16936/) (2025)
- [A Spatial-Spectral-Frequency Interactive Network for Multimodal Remote Sensing Classification](https://mlsanity.com/paper/2510.04628/) (2025)

## Abstract (arXiv)

Masked image modeling (MIM) is a highly popular and effective self-supervised learning method for image understanding. Existing MIM-based methods mostly focus on spatial feature modeling, neglecting spectral feature modeling. Meanwhile, existing MIM-based methods use Transformer for feature extraction, some local or high-frequency information may get lost. To this end, we propose a spatial-spectral masked auto-encoder (SS-MAE) for HSI and LiDAR/SAR data joint classification. Specifically, SS-MAE consists of a spatial-wise branch and a spectral-wise branch. The spatial-wise branch masks random patches and reconstructs missing pixels, while the spectral-wise branch masks random spectral channels and reconstructs missing channels. Our SS-MAE fully exploits the spatial and spectral representations of the input data. Furthermore, to complement local features in the training stage, we add two lightweight CNNs for feature extraction. Both global and local features are taken into account for feature modeling. To demonstrate the effectiveness of the proposed SS-MAE, we conduct extensive experiments on three publicly available datasets. Extensive experiments on three multi-source datasets verify the superiority of our SS-MAE compared with several state-of-the-art baselines. The source codes are available at urlhttps://github.com/summitgao/SS-MAE.