# LWM-Temporal: Sparse Spatio-Temporal Attention for Wireless Channel Representation Learning

arXiv: https://arxiv.org/abs/2603.10024 · HTML: https://mlsanity.com/paper/2603.10024/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 26 references that are in the map.
- First posted on arXiv 22 Feb 2026.

## Builds on (most cited first, 16 of 16)

- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Efficient Transformers: A Survey](https://mlsanity.com/paper/2009.06732/) (2020)
- [Efficient Content-Based Sparse Attention with Routing Transformers](https://mlsanity.com/paper/2003.05997/) (2020)
- [The Roadmap to 6G -- AI Empowered Wireless Networks](https://mlsanity.com/paper/1904.11686/) (2019)
- [DeepMIMO: A Generic Deep Learning Dataset for Millimeter Wave and Massive MIMO Applications](https://mlsanity.com/paper/1902.06435/) (2019)
- [LLM4CP: Adapting Large Language Models for Channel Prediction](https://mlsanity.com/paper/2406.14440/) (2024)
- [Large Wireless Model (LWM): A Foundation Model for Wireless Channels](https://mlsanity.com/paper/2411.08872/) (2024)
- [WiFo: Wireless Foundation Model for Channel Prediction](https://mlsanity.com/paper/2412.08908/) (2024)
- [Comparing Stochastic and Ray-tracing Datasets in Machine Learning for Wireless Applications](https://mlsanity.com/paper/2512.12449/) (2025)

## Built on by (most cited first, 5 of 5)

- [Adaptive 3D-RoPE: Physics-Aligned Rotary Positional Encoding for Wireless Foundation Models](https://mlsanity.com/paper/2605.00968/) (2026)
- [RFPrompt: Prompt-Based Expert Adaptation of the Large Wireless Model for Modulation Classification](https://mlsanity.com/paper/2605.03279/) (2026)
- [PilotWiMAE: Pilot-Native Representation Learning for Wireless Channels](https://mlsanity.com/paper/2605.22856/) (2026)
- [LWM-CDE: A Representation Space for Wireless Data Reasoning and Transferability](https://mlsanity.com/paper/2605.24077/) (2026)
- [Wireless Foundation Models: State-of-the-Art and Open Challenges](https://mlsanity.com/paper/2609.04707/) (2026)

## Abstract (arXiv)

LWM-Temporal is a new member of the Large Wireless Models (LWM) family that targets the spatiotemporal nature of wireless channels. Designed as a task-agnostic foundation model, LWM-Temporal learns universal channel embeddings that capture mobility-induced evolution and are reusable across various downstream tasks. To achieve this objective, LWM-Temporal operates in the angle-delay-time domain and introduces Sparse Spatio-Temporal Attention (SSTA), a propagation-aligned attention mechanism that restricts interactions to physically plausible neighborhoods, reducing attention complexity by an order of magnitude while preserving geometry-consistent dependencies. LWM-Temporal is pretrained in a self-supervised manner using a physics-informed masking curriculum that emulates realistic occlusions, pilot sparsity, and measurement impairments. Experimental results on channel prediction across multiple mobility regimes show consistent improvements over strong baselines, particularly under long horizons and limited fine-tuning data, highlighting the importance of geometry-aware architectures and geometry-consistent pretraining for learning transferable spatiotemporal wireless representations.