# Multimodal Transformers for Wireless Communications: A Case Study in Beam Prediction

arXiv: https://arxiv.org/abs/2309.11811 · HTML: https://mlsanity.com/paper/2309.11811/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 21 references that are in the map.
- First posted on arXiv 21 Sep 2023.

## Builds on (most cited first, 13 of 13)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Multi-Modal Fusion Transformer for End-to-End Autonomous Driving](https://mlsanity.com/paper/2104.09224/) (2021)
- [Learning Enriched Features for Fast Image Restoration and Enhancement](https://mlsanity.com/paper/2205.01649/) (2022)
- [DeepSense 6G: A Large-Scale Real-World Multi-Modal Sensing and Communication Dataset](https://mlsanity.com/paper/2211.09769/) (2022)
- [BatchFormer: Learning to Explore Sample Relationships for Robust Representation Learning](https://mlsanity.com/paper/2203.01522/) (2022)
- [Vision-Position Multi-Modal Beam Prediction Using Real Millimeter Wave Datasets](https://mlsanity.com/paper/2111.07574/) (2021)
- [LiDAR Aided Future Beam Prediction in Real-World Millimeter Wave V2I Communications](https://mlsanity.com/paper/2203.05548/) (2022)
- [Radar Aided 6G Beam Prediction: Deep Learning Algorithms and Real-World Demonstration](https://mlsanity.com/paper/2111.09676/) (2021)
- [Position Aided Beam Prediction in the Real World: How Useful GPS Locations Actually Are?](https://mlsanity.com/paper/2205.09054/) (2022)
- [Computer Vision Aided Beam Tracking in A Real-World Millimeter Wave Deployment](https://mlsanity.com/paper/2111.14803/) (2021)
- [Multi-Modal Beam Prediction Challenge 2022: Towards Generalization](https://mlsanity.com/paper/2209.07519/) (2022)
- [Applying Deep-Learning-Based Computer Vision to Wireless Communications: Methodologies, Opportunities, and Challenges](https://mlsanity.com/paper/2006.05782/) (2020)
- [When Wireless Communications Meet Computer Vision in Beyond 5G](https://mlsanity.com/paper/2010.06188/) (2020)

## Built on by (most cited first, 8 of 8)

- [WirelessGPT: A Generative Pre-trained Multi-task Learning Framework for Wireless Communication](https://mlsanity.com/paper/2502.06877/) (2025)
- [Multi-Modal Transformer and Reinforcement Learning-based Beam Management](https://mlsanity.com/paper/2410.19859/) (2024)
- [Resource-Efficient Beam Prediction in mmWave Communications with Multimodal Realistic Simulation Framework](https://mlsanity.com/paper/2504.05187/) (2025)
- [Generative AI Empowered LiDAR Point Cloud Generation with Multimodal Transformer](https://mlsanity.com/paper/2406.18542/) (2024)
- [From Large AI Models to Agentic AI: A Tutorial on Future Intelligent Communications](https://mlsanity.com/paper/2505.22311/) (2025)
- [Vehicle-to-Infrastructure Collaborative Spatial Perception via Multimodal Large Language Models](https://mlsanity.com/paper/2509.03837/) (2025)
- [Agentic AI for Embodied-enhanced Beam Prediction in Low-Altitude Economy Networks](https://mlsanity.com/paper/2603.11392/) (2026)
- [Robust Beam Prediction for V2X Networks with Multi-Modal Sensing](https://mlsanity.com/paper/2609.10200/) (2026)

## Abstract (arXiv)

Wireless communications at high-frequency bands with large antenna arrays face challenges in beam management, which can potentially be improved by multimodality sensing information from cameras, LiDAR, radar, and GPS. In this paper, we present a multimodal transformer deep learning framework for sensing-assisted beam prediction. We employ a convolutional neural network to extract the features from a sequence of images, point clouds, and radar raw data sampled over time. At each convolutional layer, we use transformer encoders to learn the hidden relations between feature tokens from different modalities and time instances over abstraction space and produce encoded vectors for the next-level feature extraction. We train the model on a combination of different modalities with supervised learning. We try to enhance the model over imbalanced data by utilizing focal loss and exponential moving average. We also evaluate data processing and augmentation techniques such as image enhancement, segmentation, background filtering, multimodal data flipping, radar signal transformation, and GPS angle calibration. Experimental results show that our solution trained on image and GPS data produces the best distance-based accuracy of predicted beams at 78.44%, with effective generalization to unseen day scenarios near 73% and night scenarios over 84%. This outperforms using other modalities and arbitrary data processing techniques, which demonstrates the effectiveness of transformers with feature fusion in performing radio beam prediction from images and GPS. Furthermore, our solution could be pretrained from large sequences of multimodality wireless data, on fine-tuning for multiple downstream radio network tasks.