# Radio Foundation Models: Pre-training Transformers for 5G-based Indoor Localization

arXiv: https://arxiv.org/abs/2410.00617 · HTML: https://mlsanity.com/paper/2410.00617/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 30 references that are in the map.
- First posted on arXiv 1 Oct 2024.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2301.08243/) (2023)
- [iBOT: Image BERT Pre-Training with Online Tokenizer](https://mlsanity.com/paper/2111.07832/) (2021)
- [Self-supervised Learning: Generative or Contrastive](https://mlsanity.com/paper/2006.08218/) (2020)
- [Self-Supervised Time Series Representation Learning via Cross Reconstruction Transformer](https://mlsanity.com/paper/2205.09928/) (2022)
- [Transformers with convolutional context for ASR](https://mlsanity.com/paper/1904.11660/) (2019)
- [Unsupervised Representation Learning for Time Series: A Review](https://mlsanity.com/paper/2308.01578/) (2023)
- [Self-Supervised and Invariant Representations for Wireless Localization](https://mlsanity.com/paper/2302.07000/) (2023)
- [Angle-Delay Profile-Based and Timestamp-Aided Dissimilarity Metrics for Channel Charting](https://mlsanity.com/paper/2308.09539/) (2023)
- [Indoor Localization with Robust Global Channel Charting: A Time-Distance-Based Approach](https://mlsanity.com/paper/2210.06294/) (2022)
- [Auxiliary Cross-Modal Representation Learning with Triplet Loss Functions for Online Handwriting Recognition](https://mlsanity.com/paper/2202.07901/) (2022)
- [Velocity-Based Channel Charting with Spatial Distribution Map Matching](https://mlsanity.com/paper/2311.08016/) (2023)
- [Positioning in 5G networks](https://mlsanity.com/paper/2102.03361/) (2021)

## Built on by (most cited first, 10 of 10)

- [WirelessGPT: A Generative Pre-trained Multi-task Learning Framework for Wireless Communication](https://mlsanity.com/paper/2502.06877/) (2025)
- [A Multi-Task Foundation Model for Wireless Channel Representation Using Contrastive and Masked Autoencoder Learning](https://mlsanity.com/paper/2505.09160/) (2025)
- [Large Wireless Localization Model (LWLM): A Foundation Model for Positioning in 6G Networks](https://mlsanity.com/paper/2505.10134/) (2025)
- [A MIMO Wireless Channel Foundation Model via CIR-CSI Consistency](https://mlsanity.com/paper/2502.11965/) (2025)
- [Multi-Modal Data-Enhanced Foundation Models for Prediction and Control in Wireless Networks: A Survey](https://mlsanity.com/paper/2601.03181/) (2026)
- [Simplicity is Key: An Unsupervised Pretraining Approach for Sparse Radio Channels](https://mlsanity.com/paper/2505.13055/) (2025)
- [Decision Feedback In-Context Learning for Wireless Symbol Detection](https://mlsanity.com/paper/2503.16594/) (2025)
- [Masked Symbol Modeling for Demodulation of Oversampled Baseband Communication Signals in Impulsive Noise-Dominated Channels](https://mlsanity.com/paper/2512.01428/) (2025)
- [OmniLoc: A Geometry-Aware Foundation Model for Anchor-Free UE Localization Across Diverse Indoor Environments](https://mlsanity.com/paper/2606.11490/) (2026)
- [Wireless Foundation Models: State-of-the-Art and Open Challenges](https://mlsanity.com/paper/2609.04707/) (2026)

## Abstract (arXiv)

Artificial Intelligence (AI)-based radio fingerprinting (FP) outperforms classic localization methods in propagation environments with strong multipath effects. However, the model and data orchestration of FP are time-consuming and costly, as it requires many reference positions and extensive measurement campaigns for each environment. Instead, modern unsupervised and self-supervised learning schemes require less reference data for localization, but either their accuracy is low or they require additional sensor information, rendering them impractical. In this paper we propose a self-supervised learning framework that pre-trains a general transformer (TF) neural network on 5G channel measurements that we collect on-the-fly without expensive equipment. Our novel pretext task randomly masks and drops input information to learn to reconstruct it. So, it implicitly learns the spatiotemporal patterns and information of the propagation environment that enable FP-based localization. Most interestingly, when we optimize this pre-trained model for localization in a given environment, it achieves the accuracy of state-of-the-art methods but requires ten times less reference data and significantly reduces the time from training to operation.