# Self-Supervised Radio Pre-training: Toward Foundational Models for Spectrogram Learning

arXiv: https://arxiv.org/abs/2411.09849 · HTML: https://mlsanity.com/paper/2411.09849/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 18 references that are in the map.
- First posted on arXiv 14 Nov 2024.

## Builds on (most cited first, 9 of 9)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [Self-supervised Learning: Generative or Contrastive](https://mlsanity.com/paper/2006.08218/) (2020)
- [An Introduction to Deep Learning for the Physical Layer](https://mlsanity.com/paper/1702.00832/) (2017)
- [Self-supervised Pretraining of Visual Features in the Wild](https://mlsanity.com/paper/2103.01988/) (2021)
- [Large Generative AI Models for Telecom: The Next Big Thing?](https://mlsanity.com/paper/2306.10249/) (2023)
- [Masked Spectrogram Prediction For Self-Supervised Audio Pre-Training](https://mlsanity.com/paper/2204.12768/) (2022)
- [Waveform Learning for Next-Generation Wireless Communication Systems](https://mlsanity.com/paper/2109.00998/) (2021)

## Built on by (most cited first, 5 of 5)

- [6G WavesFM: A Foundation Model for Sensing, Communication, and Localization](https://mlsanity.com/paper/2504.14100/) (2025)
- [Building 6G Radio Foundation Models with Transformer Architectures](https://mlsanity.com/paper/2411.09996/) (2024)
- [IQFM A Wireless Foundational Model for I/Q Streams in AI-Native 6G](https://mlsanity.com/paper/2506.06718/) (2025)
- [Multimodal Wireless Foundation Models](https://mlsanity.com/paper/2511.15162/) (2025)
- [Wireless Foundation Models: State-of-the-Art and Open Challenges](https://mlsanity.com/paper/2609.04707/) (2026)

## Abstract (arXiv)

Foundational deep learning (DL) models are general models, trained on large, diverse, and unlabelled datasets, typically using self-supervised learning techniques have led to significant advancements especially in natural language processing. These pretrained models can be fine-tuned for related downstream tasks, offering faster development and reduced training costs, while often achieving improved performance. In this work, we introduce Masked Spectrogram Modeling, a novel self-supervised learning approach for pretraining foundational DL models on radio signals. Adopting a Convolutional LSTM architecture for efficient spatio-temporal processing, we pretrain the model with an unlabelled radio dataset collected from over-the-air measurements. Subsequently, the pretrained model is fine-tuned for two downstream tasks: spectrum forecasting and segmentation. Experimental results demonstrate that our methodology achieves competitive performance in both forecasting accuracy and segmentation, validating its effectiveness for developing foundational radio models.