# A Multi-Task Foundation Model for Wireless Channel Representation Using Contrastive and Masked Autoencoder Learning

arXiv: https://arxiv.org/abs/2505.09160 · HTML: https://mlsanity.com/paper/2505.09160/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 48 references that are in the map.
- First posted on arXiv 14 May 2025.

## Builds on (most cited first, 30 of 35)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [DeepMIMO: A Generic Deep Learning Dataset for Millimeter Wave and Massive MIMO Applications](https://mlsanity.com/paper/1902.06435/) (2019)
- [Deep Learning-Based Channel Estimation](https://mlsanity.com/paper/1810.05893/) (2018)
- [Robust Contrastive Learning Using Negative Samples with Diminished Semantics](https://mlsanity.com/paper/2110.14189/) (2021)
- [LLM4CP: Adapting Large Language Models for Channel Prediction](https://mlsanity.com/paper/2406.14440/) (2024)
- [Large Wireless Model (LWM): A Foundation Model for Wireless Channels](https://mlsanity.com/paper/2411.08872/) (2024)
- [WirelessGPT: A Generative Pre-trained Multi-task Learning Framework for Wireless Communication](https://mlsanity.com/paper/2502.06877/) (2025)
- [Convolutional Neural Network based Multiple-Rate Compressive Sensing for Massive MIMO CSI Feedback: Design, Simulation, and Analysis](https://mlsanity.com/paper/1906.06007/) (2019)
- [A Tutorial on Beam Management for 3GPP NR at mmWave Frequencies](https://mlsanity.com/paper/1804.01908/) (2018)
- [WiFo: Wireless Foundation Model for Channel Prediction](https://mlsanity.com/paper/2412.08908/) (2024)
- [End-to-End Learning of Communications Systems Without a Channel Model](https://mlsanity.com/paper/1804.02276/) (2018)
- [Generating High Dimensional User-Specific Wireless Channels using Diffusion Models](https://mlsanity.com/paper/2409.03924/) (2024)
- [Generative Neural Network Channel Modeling for Millimeter-Wave UAV Communication](https://mlsanity.com/paper/2012.09133/) (2020)
- [Towards a Wireless Physical-Layer Foundation Model: Challenges and Strategies](https://mlsanity.com/paper/2403.12065/) (2024)
- [Radio Foundation Models: Pre-training Transformers for 5G-based Indoor Localization](https://mlsanity.com/paper/2410.00617/) (2024)
- [A MIMO Wireless Channel Foundation Model via CIR-CSI Consistency](https://mlsanity.com/paper/2502.11965/) (2025)
- [Building 6G Radio Foundation Models with Transformer Architectures](https://mlsanity.com/paper/2411.09996/) (2024)
- [ChannelGPT: A Large Model to Generate Digital Twin Channel for 6G Environment Intelligence](https://mlsanity.com/paper/2410.13379/) (2024)
- [End-to-End Deep Learning for TDD MIMO Systems in the 6G Upper Midbands](https://mlsanity.com/paper/2402.01033/) (2024)
- [Blockage Prediction Using Wireless Signatures: Deep Learning Enables Real-World Demonstration](https://mlsanity.com/paper/2111.08242/) (2021)
- [Grid-Free MIMO Beam Alignment through Site-Specific Deep Learning](https://mlsanity.com/paper/2209.08198/) (2022)

## Built on by (most cited first, 17 of 17)

- [Generative Diffusion Models for Wireless Networks: Fundamental, Architecture, and State-of-the-Art](https://mlsanity.com/paper/2507.16733/) (2025)
- [HeterCSI: Channel-Adaptive Heterogeneous CSI Pretraining Framework for Generalized Wireless Foundation Models](https://mlsanity.com/paper/2601.18200/) (2026)
- [EMind: A Foundation Model for Multi-task Electromagnetic Signals Understanding](https://mlsanity.com/paper/2508.18785/) (2025)
- [Multi-Modal Data-Enhanced Foundation Models for Prediction and Control in Wireless Networks: A Survey](https://mlsanity.com/paper/2601.03181/) (2026)
- [AirFM-DDA: Air-Interface Foundation Model in the Delay-Doppler-Angle Domain for AI-Native 6G](https://mlsanity.com/paper/2605.00020/) (2026)
- [Simplicity is Key: An Unsupervised Pretraining Approach for Sparse Radio Channels](https://mlsanity.com/paper/2505.13055/) (2025)
- [CIG-MAE: Cross-Modal Information-Guided Masked Autoencoder for Self-Supervised WiFi Sensing](https://mlsanity.com/paper/2512.04723/) (2025)
- [Lightweight Adaptive Feature Composition for Heterogeneous Downstream Adaptation of Wireless Foundation Models](https://mlsanity.com/paper/2606.10277/) (2026)
- [Agentic Link Construction for Environment and Intent Aware 6G Communication](https://mlsanity.com/paper/2511.05094/) (2025)
- [PilotWiMAE: Pilot-Native Representation Learning for Wireless Channels](https://mlsanity.com/paper/2605.22856/) (2026)
- [JEPA for AI-Native 6G: Predictive Representations and Open Challenges](https://mlsanity.com/paper/2607.09798/) (2026)
- [WALoMA: A Multitask Wireless Foundation Model via Adaptive Low-Rank Masked Autoencoders](https://mlsanity.com/paper/2607.25763/) (2026)
- [6G Native AI and Channel Foundation Models](https://mlsanity.com/paper/2608.14591/) (2026)
- [A Comprehensive Survey of Wireless Foundation Models for AI-Native 6G Networks](https://mlsanity.com/paper/2608.14694/) (2026)
- [CM-MAE: A Physics-Guided Cross-Modal Self-Supervised Learning Framework for Vision-Wireless Applications](https://mlsanity.com/paper/2608.15972/) (2026)
- [Wireless Foundation Models: State-of-the-Art and Open Challenges](https://mlsanity.com/paper/2609.04707/) (2026)
- [Scaling of Wireless Foundation Models via Representation Diversity and Multi-Branch Architectures](https://mlsanity.com/paper/2610.04289/) (2026)

## Abstract (arXiv)

Current applications of self-supervised learning to wireless channel representation often borrow paradigms developed for text and image processing, without fully addressing the unique characteristics and constraints of wireless communications. To bridge this gap, we introduce ContraWiMAE, Wireless Contrastive Masked Autoencoder, a transformer-based foundation model that unifies masked reconstruction and masked contrastive learning for wireless channel representation. Our key innovation is a new wireless-inspired contrastive objective that exploits the inherent characteristics of wireless environment, including noise, fading, and partial observability, as natural augmentation. Through extensive evaluation on unseen scenarios and conditions, we demonstrate our method's effectiveness in multiple downstream tasks, including cross-frequency beam selection, line-of-sight detection, and channel estimation. ContraWiMAE exhibits superior linear separability and adaptability in diverse wireless environments, demonstrating exceptional data efficiency and competitive performance compared with supervised baselines under challenging conditions. Comparative evaluations against a state-of-the-art wireless channel foundation model confirm the superior performance and data efficiency of our approach, highlighting its potential as a powerful baseline for future research in self-supervised wireless channel representation learning. To foster further work in this direction, we release the model weights and training pipeline for ContraWiMAE.