# Structure-Aware Multimodal LLM Framework for Trustworthy Near-Field Beam Prediction

arXiv: https://arxiv.org/abs/2603.16143 · HTML: https://mlsanity.com/paper/2603.16143/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 36 references that are in the map.
- First posted on arXiv 17 Mar 2026.

## Builds on (most cited first, 22 of 22)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling](https://mlsanity.com/paper/2501.17811/) (2025)
- [Sionna RT: Differentiable Ray Tracing for Radio Propagation Modeling](https://mlsanity.com/paper/2303.11103/) (2023)
- [DeepSense 6G: A Large-Scale Real-World Multi-Modal Sensing and Communication Dataset](https://mlsanity.com/paper/2211.09769/) (2022)
- [A Tutorial on Extremely Large-Scale MIMO for 6G: Fundamentals, Signal Processing, and Applications](https://mlsanity.com/paper/2307.07340/) (2023)
- [Vision-Position Multi-Modal Beam Prediction Using Real Millimeter Wave Datasets](https://mlsanity.com/paper/2111.07574/) (2021)
- [LiDAR Aided Future Beam Prediction in Real-World Millimeter Wave V2I Communications](https://mlsanity.com/paper/2203.05548/) (2022)
- [Beam Prediction based on Large Language Models](https://mlsanity.com/paper/2408.08707/) (2024)
- [AI/ML for Beam Management in 5G-Advanced: A Standardization Perspective](https://mlsanity.com/paper/2309.10575/) (2023)
- [The Bridge Toward 6G: 5G-Advanced Evolution in 3GPP Release 19](https://mlsanity.com/paper/2312.15174/) (2023)
- [Multimodal-Wireless: A Large-Scale Dataset for Sensing and Communication](https://mlsanity.com/paper/2511.03220/) (2025)
- [Hierarchical Codebook based Multiuser Beam Training for Millimeter Wave Massive MIMO](https://mlsanity.com/paper/2203.06438/) (2022)
- [Near-Field Communications: A Comprehensive Survey](https://mlsanity.com/paper/2401.05900/) (2024)
- [M2BeamLLM: Multimodal Sensing-empowered mmWave Beam Prediction with Large Language Models](https://mlsanity.com/paper/2506.14532/) (2025)
- [Two-Stage Hierarchical Beam Training for Near-Field Communications](https://mlsanity.com/paper/2302.12511/) (2023)
- [Near-Field Wideband Channel Estimation for Extremely Large-Scale MIMO](https://mlsanity.com/paper/2212.08401/) (2022)
- [Keypoint Detection Empowered Near-Field User Localization and Channel Reconstruction](https://mlsanity.com/paper/2501.11844/) (2025)
- [Multi-Modal Large Models Based Beam Prediction: An Example Empowered by DeepSeek](https://mlsanity.com/paper/2506.05921/) (2025)
- [Large-Model AI for Near Field Beam Prediction: A CNN-GPT2 Framework for 6G XL-MIMO](https://mlsanity.com/paper/2510.22557/) (2025)

## Abstract (arXiv)

In near-field extremely large-scale multiple-input multiple-output (XL-MIMO) systems, spherical wavefront propagation expands the traditional beam codebook into the joint angular-distance domain, rendering conventional beam training prohibitively inefficient, especially in complex 3-dimensional (3D) low-altitude environments. Furthermore, since near-field beam variations are deeply coupled not only with user positions but also with the physical surroundings, precise beam alignment demands profound environmental understanding capabilities. To address this, we propose a large language model (LLM)-driven multimodal framework that fuses historical GPS data, RGB image, LiDAR data, and strategically designed task-specific textual prompts. By utilizing the powerful emergent reasoning and generalization capabilities of the LLM, our approach learns complex spatial dynamics to achieve superior environmental comprehension...