# PilotWiMAE: Pilot-Native Representation Learning for Wireless Channels

arXiv: https://arxiv.org/abs/2605.22856 · HTML: https://mlsanity.com/paper/2605.22856/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 44 references that are in the map.
- First posted on arXiv 19 May 2026.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [Measuring Robustness to Natural Distribution Shifts in Image Classification](https://mlsanity.com/paper/2007.00644/) (2020)
- [How Mask Matters: Towards Theoretical Understandings of Masked Autoencoders](https://mlsanity.com/paper/2210.08344/) (2022)
- [LLM4CP: Adapting Large Language Models for Channel Prediction](https://mlsanity.com/paper/2406.14440/) (2024)
- [WiFo: Wireless Foundation Model for Channel Prediction](https://mlsanity.com/paper/2412.08908/) (2024)
- [A Multi-Task Foundation Model for Wireless Channel Representation Using Contrastive and Masked Autoencoder Learning](https://mlsanity.com/paper/2505.09160/) (2025)
- [6G WavesFM: A Foundation Model for Sensing, Communication, and Localization](https://mlsanity.com/paper/2504.14100/) (2025)
- [Self-Supervised and Invariant Representations for Wireless Localization](https://mlsanity.com/paper/2302.07000/) (2023)
- [A Wireless Foundation Model for Multi-Task Prediction](https://mlsanity.com/paper/2507.05938/) (2025)
- [Large Wireless Localization Model (LWLM): A Foundation Model for Positioning in 6G Networks](https://mlsanity.com/paper/2505.10134/) (2025)
- [A MIMO Wireless Channel Foundation Model via CIR-CSI Consistency](https://mlsanity.com/paper/2502.11965/) (2025)
- [WirelessJEPA: A Multi-Antenna Foundation Model using Spatio-temporal Wireless Latent Predictions](https://mlsanity.com/paper/2601.20190/) (2026)
- [LVM4CSI: Enabling Direct Application of Pre-Trained Large Vision Models for Wireless Channel Tasks](https://mlsanity.com/paper/2507.05121/) (2025)
- [CSI-MAE: A Masked Autoencoder-based Channel Foundation Model](https://mlsanity.com/paper/2601.03789/) (2026)
- [WiFo-CF: Wireless Foundation Model for CSI Feedback](https://mlsanity.com/paper/2508.04068/) (2025)
- [WiFo-2: a generalist foundation model unifies heterogeneous wireless system design](https://mlsanity.com/paper/2511.22222/) (2025)
- [LWM-Temporal: Sparse Spatio-Temporal Attention for Wireless Channel Representation Learning](https://mlsanity.com/paper/2603.10024/) (2026)
- [Filter-and-Attend: Wireless Channel Foundation Model with Noise-Plus-Interference Suppression Structure](https://mlsanity.com/paper/2509.15993/) (2025)
- [Reducing Pilots in Channel Estimation with Predictive Foundation Models](https://mlsanity.com/paper/2512.15562/) (2025)
- [Computationally Efficient Neural Receivers via Axial Self-Attention](https://mlsanity.com/paper/2510.12941/) (2025)

## Abstract (arXiv)

Channel foundation models assume access to fully observed channels, an assumption that fails in deployment. We introduce PilotWiMAE, a self-supervised framework whose encoder ingests noisy pilot observations directly and whose attention factorizes along the axis separating temporal from joint space-frequency processing, an inductive bias inspired by the physics of the problem. Pilot input shrinks the observation space by up to two orders of magnitude and also removes the unrealistic assumption of full-CSI availability while incurring lower latency. The factorized design generates robust representations by exploiting the separable channel structure and allows a pretraining mask ratio of 99%. We pair patch-normalized reconstruction, which captures small-scale fading structure, with an auxiliary scale loss that recovers the large-scale fading features, and use an AWGN curriculum to match pilot noise at pretraining and deployment. Pretrained solely on 3.5,GHz and evaluated at 28,GHz across in-distribution and out-of-distribution settings, PilotWiMAE's cross-frequency beam selection and channel characterization beat supervised baselines despite operating on a smaller observation space. To weaken the coupling between decoder capacity and representation quality, we further propose a decoder-centric pretraining stage following the encoder-decoder joint pretraining, which allows PilotWiMAE to demonstrate competitive channel estimation without sacrificing representation quality. To foster further work in this direction, we release the PilotWiMAE pretrained weights and training pipeline, together with CSIGen, our Sionna-based ray-tracing channel-generation tool, and the channel datasets used in this work.