# LinFormer: A Linear-based Lightweight Transformer Architecture For Time-Aware MIMO Channel Prediction

arXiv: https://arxiv.org/abs/2410.21351 · HTML: https://mlsanity.com/paper/2410.21351/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 48 references that are in the map.
- First posted on arXiv 28 Oct 2024.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [Are Transformers Effective for Time Series Forecasting?](https://mlsanity.com/paper/2205.13504/) (2022)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [On the Road to 6G: Visions, Requirements, Key Technologies and Testbeds](https://mlsanity.com/paper/2302.14536/) (2023)
- [TSLANet: Rethinking Transformers for Time Series Representation Learning](https://mlsanity.com/paper/2404.08472/) (2024)
- [Massive MIMO Channel Prediction: Kalman Filtering vs. Machine Learning](https://mlsanity.com/paper/2009.09967/) (2020)
- [Unnatural Error Correction: GPT-4 Can Almost Perfectly Handle Unnatural Scrambled Text](https://mlsanity.com/paper/2311.18805/) (2023)
- [Artificial intelligence enabled radio propagation for communications-Part II: Scenario identification and channel modeling](https://mlsanity.com/paper/2111.12228/) (2021)
- [Addressing the curse of mobility in massive MIMO with Prony-based angular-delay domain channel predictions](https://mlsanity.com/paper/1912.11330/) (2019)
- [6G Wireless Channel Measurements and Models: Trends and Challenges](https://mlsanity.com/paper/2012.06381/) (2020)
- [A Comparison of Neural Networks for Wireless Channel Prediction](https://mlsanity.com/paper/2308.14020/) (2023)
- [Deep Learning-based Beam Tracking for Millimeter-wave Communications under Mobility](https://mlsanity.com/paper/2102.09785/) (2021)
- [Massive MIMO in 5G: How Beamforming, Codebooks, and Feedback Enable Larger Arrays](https://mlsanity.com/paper/2301.13390/) (2023)
- [Time-Varying Channel Prediction for RIS-Assisted MU-MISO Networks via Deep Learning](https://mlsanity.com/paper/2111.04971/) (2021)
- [C-GRBFnet: A Physics-Inspired Generative Deep Neural Network for Channel Representation and Prediction](https://mlsanity.com/paper/2112.02615/) (2021)
- [Pervasive wireless channel modeling theory and applications to 6G GBSMs for all frequency bands and all scenarios](https://mlsanity.com/paper/2206.02442/) (2022)
- [Robust Adaptive Beam Tracking for Mobile Millimetre Wave Communications](https://mlsanity.com/paper/2005.00980/) (2020)
- [Deep Learning-Empowered Predictive Beamforming for IRS-Assisted Multi-User Communications](https://mlsanity.com/paper/2104.12309/) (2021)

## Built on by (most cited first, 6 of 6)

- [AI-driven Wireless Positioning: Fundamentals, Standards, State-of-the-art, and Challenges](https://mlsanity.com/paper/2501.14970/) (2025)
- [Multi-View Learning with Context-Guided Receptance for Image Denoising](https://mlsanity.com/paper/2505.02705/) (2025)
- [Great X: A Unified Multi-Modal Simulator Bridging the Sim2Real Gap for 6G](https://mlsanity.com/paper/2507.08716/) (2025)
- [CSI Prediction Using Diffusion Models](https://mlsanity.com/paper/2510.11214/) (2025)
- [Efficient UAV trajectory prediction: A multi-modal deep diffusion framework](https://mlsanity.com/paper/2602.00107/) (2026)
- [TRACE-CRC: Trajectory-Adaptive Conformal Risk Control for Multi-Step Channel State Information Prediction](https://mlsanity.com/paper/2608.27124/) (2026)

## Abstract (arXiv)

The emergence of 6th generation (6G) mobile networks brings new challenges in supporting high-mobility communications, particularly in addressing the issue of channel aging. While existing channel prediction methods offer improved accuracy at the expense of increased computational complexity, limiting their practical application in mobile networks. To address these challenges, we present LinFormer, an innovative channel prediction framework based on a scalable, all-linear, encoder-only Transformer model. Our approach, inspired by natural language processing (NLP) models such as BERT, adapts an encoder-only architecture specifically for channel prediction tasks. We propose replacing the computationally intensive attention mechanism commonly used in Transformers with a time-aware multi-layer perceptron (TMLP), significantly reducing computational demands. The inherent time awareness of TMLP module makes it particularly suitable for channel prediction tasks. We enhance LinFormer's training process by employing a weighted mean squared error loss (WMSELoss) function and data augmentation techniques, leveraging larger, readily available communication datasets. Our approach achieves a substantial reduction in computational complexity while maintaining high prediction accuracy, making it more suitable for deployment in cost-effective base stations (BS). Comprehensive experiments using both simulated and measured data demonstrate that LinFormer outperforms existing methods across various mobility scenarios, offering a promising solution for future wireless communication systems.