# Simplicity is Key: An Unsupervised Pretraining Approach for Sparse Radio Channels

arXiv: https://arxiv.org/abs/2505.13055 · HTML: https://mlsanity.com/paper/2505.13055/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 64 references that are in the map.
- First posted on arXiv 19 May 2025.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Language Modeling with Gated Convolutional Networks](https://mlsanity.com/paper/1612.08083/) (2016)
- [Improving Dictionary Learning with Gated Sparse Autoencoders](https://mlsanity.com/paper/2404.16014/) (2024)
- [Model-Based Deep Learning](https://mlsanity.com/paper/2012.08405/) (2020)
- [DeepMIMO: A Generic Deep Learning Dataset for Millimeter Wave and Massive MIMO Applications](https://mlsanity.com/paper/1902.06435/) (2019)
- [Deep Learning in Mobile and Wireless Networking: A Survey](https://mlsanity.com/paper/1803.04311/) (2018)
- [Simplicity Bias in Transformers and their Ability to Learn Sparse Boolean Functions](https://mlsanity.com/paper/2211.12316/) (2022)
- [Channel Charting: Locating Users within the Radio Environment using Channel State Information](https://mlsanity.com/paper/1807.05247/) (2018)
- [Large Wireless Model (LWM): A Foundation Model for Wireless Channels](https://mlsanity.com/paper/2411.08872/) (2024)
- [WirelessGPT: A Generative Pre-trained Multi-task Learning Framework for Wireless Communication](https://mlsanity.com/paper/2502.06877/) (2025)
- [A Very Brief Introduction to Machine Learning With Applications to Communication Systems](https://mlsanity.com/paper/1808.02342/) (2018)
- [A Tutorial on Beam Management for 3GPP NR at mmWave Frequencies](https://mlsanity.com/paper/1804.01908/) (2018)
- [Approximating the Void: Learning Stochastic Channel Models from Observation with Variational Generative Adversarial Networks](https://mlsanity.com/paper/1805.06350/) (2018)
- [CSI-based Positioning in Massive MIMO systems using Convolutional Neural Networks](https://mlsanity.com/paper/1911.11523/) (2019)
- [A Multi-Task Foundation Model for Wireless Channel Representation Using Contrastive and Masked Autoencoder Learning](https://mlsanity.com/paper/2505.09160/) (2025)
- [Self-Supervised and Invariant Representations for Wireless Localization](https://mlsanity.com/paper/2302.07000/) (2023)
- [Large Wireless Localization Model (LWLM): A Foundation Model for Positioning in 6G Networks](https://mlsanity.com/paper/2505.10134/) (2025)
- [Radio Foundation Models: Pre-training Transformers for 5G-based Indoor Localization](https://mlsanity.com/paper/2410.00617/) (2024)
- [BERT4MIMO: A Foundation Model using BERT Architecture for Massive MIMO Channel State Information Prediction](https://mlsanity.com/paper/2501.01802/) (2025)
- [Indoor Localization with Robust Global Channel Charting: A Time-Distance-Based Approach](https://mlsanity.com/paper/2210.06294/) (2022)
- [CSI-fingerprinting Indoor Localization via Attention-Augmented Residual Convolutional Neural Network](https://mlsanity.com/paper/2205.05775/) (2022)
- [Towards Practical Indoor Positioning Based on Massive MIMO Systems](https://mlsanity.com/paper/1905.11858/) (2019)
- [Attention Aided CSI Wireless Localization](https://mlsanity.com/paper/2203.10506/) (2022)
- [Sharp Convergence Rates for Matching Pursuit](https://mlsanity.com/paper/2307.07679/) (2023)

## Built on by (most cited first, 1 of 1)

- [Wireless Foundation Models: State-of-the-Art and Open Challenges](https://mlsanity.com/paper/2609.04707/) (2026)

## Abstract (arXiv)

Unsupervised representation learning for wireless channel state information (CSI)reduces reliance on labeled data, thereby lowering annotation costs, and often improves performance on downstream tasks. However, state-of-the-art approaches take little or no account of domain-specific knowledge, forcing the model to learn well-known concepts solely from data. We introduce Sparse pretrained Radio Transformer (SpaRTran), a hybrid method based on the concept of compressed sensing for wireless channels. In contrast to existing work, SpaRTran builds around a wireless channel model that constrains the optimization procedure to physically meaningful solutions and induces a strong inductive bias. Compared to the state of the art, SpaRTran cuts positioning error by up to 28% and increases top-1 codebook selection accuracy for beamforming by 26 percentage points. Our results show that capturing the sparse nature of radio propagation as an unsupervised learning objective improves performance for network optimization and radio-localization tasks.