# HeterCSI: Channel-Adaptive Heterogeneous CSI Pretraining Framework for Generalized Wireless Foundation Models

arXiv: https://arxiv.org/abs/2601.18200 · HTML: https://mlsanity.com/paper/2601.18200/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 49 references that are in the map.
- First posted on arXiv 26 Jan 2026.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [Train faster, generalize better: Stability of stochastic gradient descent](https://mlsanity.com/paper/1509.01240/) (2015)
- [From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning](https://mlsanity.com/paper/2308.12032/) (2023)
- [SpectralGPT: Spectral Remote Sensing Foundation Model](https://mlsanity.com/paper/2311.07113/) (2023)
- [LLM4CP: Adapting Large Language Models for Channel Prediction](https://mlsanity.com/paper/2406.14440/) (2024)
- [Large Wireless Model (LWM): A Foundation Model for Wireless Channels](https://mlsanity.com/paper/2411.08872/) (2024)
- [WirelessGPT: A Generative Pre-trained Multi-task Learning Framework for Wireless Communication](https://mlsanity.com/paper/2502.06877/) (2025)
- [WiFo: Wireless Foundation Model for Channel Prediction](https://mlsanity.com/paper/2412.08908/) (2024)
- [Massive MIMO Channel Prediction: Kalman Filtering vs. Machine Learning](https://mlsanity.com/paper/2009.09967/) (2020)
- [A Multi-Task Foundation Model for Wireless Channel Representation Using Contrastive and Masked Autoencoder Learning](https://mlsanity.com/paper/2505.09160/) (2025)
- [LLM4WM: Adapting LLM for Wireless Multi-Tasking](https://mlsanity.com/paper/2501.12983/) (2025)
- [Domain Generalization in Machine Learning Models for Wireless Communications: Concepts, State-of-the-Art, and Open Issues](https://mlsanity.com/paper/2303.08106/) (2023)
- [Large Wireless Localization Model (LWLM): A Foundation Model for Positioning in 6G Networks](https://mlsanity.com/paper/2505.10134/) (2025)
- [Addressing the curse of mobility in massive MIMO with Prony-based angular-delay domain channel predictions](https://mlsanity.com/paper/1912.11330/) (2019)
- [BERT4MIMO: A Foundation Model using BERT Architecture for Massive MIMO Channel State Information Prediction](https://mlsanity.com/paper/2501.01802/) (2025)
- [ChannelGPT: A Large Model to Generate Digital Twin Channel for 6G Environment Intelligence](https://mlsanity.com/paper/2410.13379/) (2024)
- [LVM4CSI: Enabling Direct Application of Pre-Trained Large Vision Models for Wireless Channel Tasks](https://mlsanity.com/paper/2507.05121/) (2025)
- [Enabling 6G Through Multi-Domain Channel Extrapolation: Opportunities and Challenges of Generative Artificial Intelligence](https://mlsanity.com/paper/2509.01125/) (2025)
- [LLM4PG: Adapting Large Language Model for Pathloss Map Generation via Synesthesia of Machines](https://mlsanity.com/paper/2511.02423/) (2025)
- [Sharp Minima Can Generalize: A Loss Landscape Perspective On Data](https://mlsanity.com/paper/2511.04808/) (2025)
- [Bridging the Modality Gap: Enhancing Channel Prediction with Semantically Aligned LLMs and Knowledge Distillation](https://mlsanity.com/paper/2505.12729/) (2025)
- [MMSE Channel Estimation in Large-Scale MIMO: Improved Robustness with Reduced Complexity](https://mlsanity.com/paper/2404.03279/) (2024)
- [WiFo-M2: Empower Wireless Communications With Plug-and-Play Environment Sensing via Foundation Model](https://mlsanity.com/paper/2601.09179/) (2026)

## Built on by (most cited first, 4 of 4)

- [AirFM-DDA: Air-Interface Foundation Model in the Delay-Doppler-Angle Domain for AI-Native 6G](https://mlsanity.com/paper/2605.00020/) (2026)
- [Adaptive 3D-RoPE: Physics-Aligned Rotary Positional Encoding for Wireless Foundation Models](https://mlsanity.com/paper/2605.00968/) (2026)
- [Towards CSI-Native Foundation Models: A Channel-Adaptive Roadmap for 6G](https://mlsanity.com/paper/2606.20670/) (2026)
- [Wireless Foundation Models: State-of-the-Art and Open Challenges](https://mlsanity.com/paper/2609.04707/) (2026)

## Abstract (arXiv)

Wireless foundation models promise transformative capabilities for channel state information (CSI) processing across diverse 6G network applications, yet face fundamental challenges due to the inherent dual heterogeneity of CSI across both scale and scenario dimensions. However, current pretraining approaches either constrain inputs to fixed dimensions or isolate training by scale, limiting the generalization and scalability of wireless foundation models. In this paper, we propose HeterCSI, a channel-adaptive pretraining framework that reconciles training efficiency with robust cross-scenario generalization via a new understanding of gradient dynamics in heterogeneous CSI pretraining. Our key insight reveals that CSI scale heterogeneity primarily causes destructive gradient interference, while scenario diversity actually promotes constructive gradient alignment when properly managed. Specifically, we formulate heterogeneous CSI batch construction as a partitioning optimization problem that minimizes zero-padding overhead while preserving scenario diversity. To solve this, we develop a scale-aware adaptive batching strategy that aligns CSI samples of similar scales, and design a double-masking mechanism to isolate valid signals from padding artifacts. Extensive experiments on 12 datasets demonstrate that HeterCSI establishes a generalized foundation model without scenario-specific finetuning, achieving superior average performance over full-shot baselines. Compared to the state-of-the-art zero-shot benchmark WiFo, it reduces NMSE by 7.19 dB, 4.08 dB, and 5.27 dB for CSI reconstruction, time-domain, and frequency-domain prediction, respectively. The proposed HeterCSI framework also reduces training latency by 53% compared to existing approaches while improving generalization performance by 1.53 dB on average.