# Dual-Domain Masked Image Modeling: A Self-Supervised Pretraining Strategy Using Spatial and Frequency Domain Masking for Hyperspectral Data

arXiv: https://arxiv.org/abs/2505.03220 · HTML: https://mlsanity.com/paper/2505.03220/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 23 references that are in the map.
- First posted on arXiv 6 May 2025.

## Builds on (most cited first, 11 of 11)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [SpectralFormer: Rethinking Hyperspectral Image Classification with Transformers](https://mlsanity.com/paper/2107.02988/) (2021)
- [Graph Convolutional Networks for Hyperspectral Image Classification](https://mlsanity.com/paper/2008.02457/) (2020)
- [Masked Frequency Modeling for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2206.07706/) (2022)
- [Trends in deep learning for medical hyperspectral image analysis](https://mlsanity.com/paper/2011.13974/) (2020)
- [Pre-training with Random Orthogonal Projection Image Modeling](https://mlsanity.com/paper/2310.18737/) (2023)
- [FactoFormer: Factorized Hyperspectral Transformers with Self-Supervised Pretraining](https://mlsanity.com/paper/2309.09431/) (2023)
- [Spatioformer: A Geo-encoded Transformer for Large-Scale Plant Species Richness Prediction](https://mlsanity.com/paper/2410.19256/) (2024)
- [Spectral-Enhanced Transformers: Leveraging Large-Scale Pretrained Models for Hyperspectral Object Tracking](https://mlsanity.com/paper/2502.18748/) (2025)

## Abstract (arXiv)

Hyperspectral images (HSIs) capture rich spectral signatures that reveal vital material properties, offering broad applicability across various domains. However, the scarcity of labeled HSI data limits the full potential of deep learning, especially for transformer-based architectures that require large-scale training. To address this constraint, we propose Spatial-Frequency Masked Image Modeling (SFMIM), a self-supervised pretraining strategy for hyperspectral data that utilizes the large portion of unlabeled data. Our method introduces a novel dual-domain masking mechanism that operates in both spatial and frequency domains. The input HSI cube is initially divided into non-overlapping patches along the spatial dimension, with each patch comprising the entire spectrum of its corresponding spatial location. In spatial masking, we randomly mask selected patches and train the model to reconstruct the masked inputs using the visible patches. Concurrently, in frequency masking, we remove portions of the frequency components of the input spectra and predict the missing frequencies. By learning to reconstruct these masked components, the transformer-based encoder captures higher-order spectral-spatial correlations. We evaluate our approach on three publicly available HSI classification benchmarks and demonstrate that it achieves state-of-the-art performance. Notably, our model shows rapid convergence during fine-tuning, highlighting the efficiency of our pretraining strategy.