# CSI-BERT2: A BERT-inspired Framework for Efficient CSI Prediction and Classification in Wireless Communication and Sensing

arXiv: https://arxiv.org/abs/2412.06861 · HTML: https://mlsanity.com/paper/2412.06861/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 67 references that are in the map.
- First posted on arXiv 9 Dec 2024.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Are Transformers Effective for Time Series Forecasting?](https://mlsanity.com/paper/2205.13504/) (2022)
- [Variable Rate Image Compression with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06085/) (2015)
- [Sionna: An Open-Source Library for Next-Generation Physical Layer Research](https://mlsanity.com/paper/2203.11854/) (2022)
- [Edge Learning for B5G Networks with Distributed Signal Processing: Semantic Communication, Edge Computing, and Wireless Sensing](https://mlsanity.com/paper/2206.00422/) (2022)
- [Task-Oriented Sensing, Computation, and Communication Integration for Multi-Device Edge AI](https://mlsanity.com/paper/2207.00969/) (2022)
- [MetaFi: Device-Free Pose Estimation via Commodity WiFi for Metaverse Avatar Simulation](https://mlsanity.com/paper/2208.10414/) (2022)
- [An overview of domain-specific foundation model: key technologies, applications and challenges](https://mlsanity.com/paper/2409.04267/) (2024)
- [Finding the Missing Data: A BERT-inspired Approach Against Package Loss in Wireless Sensing](https://mlsanity.com/paper/2403.12400/) (2024)
- [Rethinking Resource Management in Edge Learning: A Joint Pre-training and Fine-tuning Design Paradigm](https://mlsanity.com/paper/2404.00836/) (2024)
- [Integrated Sensing, Communication, and Computation Over-the-Air: MIMO Beamforming Design](https://mlsanity.com/paper/2201.12581/) (2022)
- [CrossFi: A Cross Domain Wi-Fi Sensing Framework Based on Siamese Network](https://mlsanity.com/paper/2408.10919/) (2024)
- [Deep Multimodal Learning: Merging Sensory Data for Massive MIMO Channel Prediction](https://mlsanity.com/paper/2007.09366/) (2020)
- [Defining error accumulation in ML atmospheric simulators](https://mlsanity.com/paper/2405.14714/) (2024)
- [AI Generated Signal for Wireless Sensing](https://mlsanity.com/paper/2312.14563/) (2023)
- [Modelling the 5G Energy Consumption using Real-world Data: Energy Fingerprint is All You Need](https://mlsanity.com/paper/2406.16929/) (2024)
- [Acquisition of Channel State Information for mmWave Massive MIMO: Traditional and Machine Learning-based Approaches](https://mlsanity.com/paper/2006.08894/) (2020)
- [3D Human Mesh Construction Leveraging Wi-Fi](https://mlsanity.com/paper/2210.10957/) (2022)
- [LoFi: Vision-Aided Label Generator for Wi-Fi Localization and Tracking](https://mlsanity.com/paper/2412.05074/) (2024)

## Built on by (most cited first, 11 of 11)

- [Bridging MARL to SARL: An Order-Independent Multi-Agent Transformer via Latent Consensus](https://mlsanity.com/paper/2604.13472/) (2026)
- [Lightweight Adaptive Feature Composition for Heterogeneous Downstream Adaptation of Wireless Foundation Models](https://mlsanity.com/paper/2606.10277/) (2026)
- [Label Unbalance in High-frequency Trading](https://mlsanity.com/paper/2503.09988/) (2025)
- [A Short Overview of Multi-Modal Wi-Fi Sensing](https://mlsanity.com/paper/2505.06682/) (2025)
- [Transforming Decoder-Only Transformers for Accurate WiFi-Telemetry Based Indoor Localization](https://mlsanity.com/paper/2505.15835/) (2025)
- [CSI Prediction Using Diffusion Models](https://mlsanity.com/paper/2510.11214/) (2025)
- [CSI-4CAST: A Hybrid Deep Learning Model for CSI Prediction with Comprehensive Robustness and Generalization Testing](https://mlsanity.com/paper/2510.12996/) (2025)
- [Cross-Domain Multi-Person Human Activity Recognition via Near-Field Wi-Fi Sensing](https://mlsanity.com/paper/2510.17816/) (2025)
- [FuseFi: Combining Irregularly Sampled CSI from Diverse Communication Packets and Frequency Bands for Wi-Fi Sensing](https://mlsanity.com/paper/2512.22143/) (2025)
- [AutoFed: Personalized Federated Traffic Prediction via Adaptive Prompt](https://mlsanity.com/paper/2512.24625/) (2025)
- [Variational Dual-path Attention Network for CSI-Based Gesture Recognition](https://mlsanity.com/paper/2601.13745/) (2026)

## Abstract (arXiv)

Channel state information (CSI) is a fundamental component in both wireless communication and sensing systems, enabling critical functions such as radio resource optimization and environmental perception. In wireless sensing, data scarcity and packet loss hinder efficient model training, while in wireless communication, high-dimensional CSI matrices and short coherent times caused by high mobility present challenges in CSI estimation. To address these issues, we propose a unified framework named CSI-BERT2 for CSI prediction and classification tasks, built on CSI-BERT, which adapts BERT to capture the complex relationships among CSI sequences through a bidirectional self-attention mechanism. We introduce a two-stage training method that first uses a mask language model (MLM) to enable the model to learn general feature extraction from scarce datasets in an unsupervised manner, followed by fine-tuning for specific downstream tasks. Specifically, we extend MLM into a mask prediction model (MPM), which efficiently addresses the CSI prediction task. To further enhance the representation capacity of CSI data, we modify the structure of the original CSI-BERT. We introduce an adaptive re-weighting layer (ARL) to enhance subcarrier representation and a multi-layer perceptron (MLP)-based temporal embedding module to mitigate temporal information loss problem inherent in the original Transformer. Extensive experiments on both real-world collected and simulated datasets demonstrate that CSI-BERT2 achieves state-of-the-art performance across all tasks. Our results further show that CSI-BERT2 generalizes effectively across varying sampling rates and robustly handles discontinuous CSI sequences caused by packet loss-challenges that conventional methods fail to address. The dataset and code are publicly available at https://github.com/RS2002/CSI-BERT2 .