# Building 6G Radio Foundation Models with Transformer Architectures

arXiv: https://arxiv.org/abs/2411.09996 · HTML: https://mlsanity.com/paper/2411.09996/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 14 references that are in the map.
- First posted on arXiv 15 Nov 2024.

## Builds on (most cited first, 11 of 11)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Self-supervised Learning: Generative or Contrastive](https://mlsanity.com/paper/2006.08218/) (2020)
- [Self-supervised Pretraining of Visual Features in the Wild](https://mlsanity.com/paper/2103.01988/) (2021)
- [Large Generative AI Models for Telecom: The Next Big Thing?](https://mlsanity.com/paper/2306.10249/) (2023)
- [EfficientFi: Towards Large-Scale Lightweight WiFi Sensing via CSI Compression](https://mlsanity.com/paper/2204.04138/) (2022)
- [Towards a Wireless Physical-Layer Foundation Model: Challenges and Strategies](https://mlsanity.com/paper/2403.12065/) (2024)
- [Waveform Learning for Next-Generation Wireless Communication Systems](https://mlsanity.com/paper/2109.00998/) (2021)
- [Revolutionizing Wireless Networks with Self-Supervised Learning: A Pathway to Intelligent Communications](https://mlsanity.com/paper/2406.06872/) (2024)
- [Self-Supervised Radio Pre-training: Toward Foundational Models for Spectrogram Learning](https://mlsanity.com/paper/2411.09849/) (2024)

## Built on by (most cited first, 9 of 9)

- [A Multi-Task Foundation Model for Wireless Channel Representation Using Contrastive and Masked Autoencoder Learning](https://mlsanity.com/paper/2505.09160/) (2025)
- [6G WavesFM: A Foundation Model for Sensing, Communication, and Localization](https://mlsanity.com/paper/2504.14100/) (2025)
- [A MIMO Wireless Channel Foundation Model via CIR-CSI Consistency](https://mlsanity.com/paper/2502.11965/) (2025)
- [IQFM A Wireless Foundational Model for I/Q Streams in AI-Native 6G](https://mlsanity.com/paper/2506.06718/) (2025)
- [Multimodal Wireless Foundation Models](https://mlsanity.com/paper/2511.15162/) (2025)
- [System One Models for Wireless Decision-Making:Applications and Performance Evaluation](https://mlsanity.com/paper/2610.04345/) (2026)
- [Internet of Everything in the 6G Era: Paradigms, Enablers, Potentials and Future Directions](https://mlsanity.com/paper/2604.25018/) (2026)
- [Wireless Foundation Models: State-of-the-Art and Open Challenges](https://mlsanity.com/paper/2609.04707/) (2026)
- [Scaling of Wireless Foundation Models via Representation Diversity and Multi-Branch Architectures](https://mlsanity.com/paper/2610.04289/) (2026)

## Abstract (arXiv)

Foundation deep learning (DL) models are general models, designed to learn general, robust and adaptable representations of their target modality, enabling finetuning across a range of downstream tasks. These models are pretrained on large, unlabeled datasets using self-supervised learning (SSL). Foundation models have demonstrated better generalization than traditional supervised approaches, a critical requirement for wireless communications where the dynamic environment demands model adaptability. In this work, we propose and demonstrate the effectiveness of a Vision Transformer (ViT) as a radio foundation model for spectrogram learning. We introduce a Masked Spectrogram Modeling (MSM) approach to pretrain the ViT in a self-supervised fashion. We evaluate the ViT-based foundation model on two downstream tasks: Channel State Information (CSI)-based Human Activity sensing and Spectrogram Segmentation. Experimental results demonstrate competitive performance to supervised training while generalizing across diverse domains. Notably, the pretrained ViT model outperforms a four-times larger model that is trained from scratch on the spectrogram segmentation task, while requiring significantly less training time, and achieves competitive performance on the CSI-based human activity sensing task. This work demonstrates the effectiveness of ViT with MSM for pretraining as a promising technique for scalable foundation model development in future 6G networks.