# Deep Learning on Multimodal Sensor Data at the Wireless Edge for Vehicular Network

arXiv: https://arxiv.org/abs/2201.04712 · HTML: https://mlsanity.com/paper/2201.04712/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 60 references that are in the map.
- First posted on arXiv 12 Jan 2022.

## Builds on (most cited first, 16 of 16)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [MFAS: Multimodal Fusion Architecture Search](https://mlsanity.com/paper/1903.06496/) (2019)
- [A Tutorial on Beam Management for 3GPP NR at mmWave Frequencies](https://mlsanity.com/paper/1804.01908/) (2018)
- [Millimeter Wave Base Stations with Cameras: Vision Aided Beam and Blockage Prediction](https://mlsanity.com/paper/1911.06255/) (2019)
- [Weather Influence and Classification with Automotive Lidar Sensors](https://mlsanity.com/paper/1906.07675/) (2019)
- [5G MIMO Data for Machine Learning: Application to Beam-Selection using Deep Learning](https://mlsanity.com/paper/2106.05370/) (2021)
- [LIDAR Data for Deep Learning-Based mmWave Beam-Selection](https://mlsanity.com/paper/1908.07488/) (2019)
- [Open-World Class Discovery with Kernel Networks](https://mlsanity.com/paper/2012.06957/) (2020)
- [ViWi Vision-Aided mmWave Beam Tracking: Dataset, Task, and Baseline Solutions](https://mlsanity.com/paper/2002.02445/) (2020)
- [Applying Deep-Learning-Based Computer Vision to Wireless Communications: Methodologies, Opportunities, and Challenges](https://mlsanity.com/paper/2006.05782/) (2020)
- [3D Scene Based Beam Selection for mmWave Communications](https://mlsanity.com/paper/1911.08409/) (2019)
- [Machine Learning on Camera Images for Fast mmWave Beamforming](https://mlsanity.com/paper/2102.07337/) (2021)
- [Standalone and Non-Standalone Beam Management for 3GPP NR at mmWaves](https://mlsanity.com/paper/1805.04268/) (2018)
- [Energy and Latency of Beamforming Architectures for Initial Access in mmWave Wireless Networks](https://mlsanity.com/paper/2001.02034/) (2020)
- [Beam Alignment and Tracking for Autonomous Vehicular Communication using IEEE 802.11ad-based Radar](https://mlsanity.com/paper/1712.02453/) (2017)

## Built on by (most cited first, 15 of 15)

- [Babel: A Scalable Pre-trained Model for Multi-Modal Sensing via Expandable Modality Alignment](https://mlsanity.com/paper/2407.17777/) (2024)
- [SL-ACC: A Communication-Efficient Split Learning Framework with Adaptive Channel-wise Compression](https://mlsanity.com/paper/2508.12984/) (2025)
- [Multiverse at the Edge: Interacting Real World and Digital Twins for Wireless Beamforming](https://mlsanity.com/paper/2305.10350/) (2023)
- [Resource-Efficient Beam Prediction in mmWave Communications with Multimodal Realistic Simulation Framework](https://mlsanity.com/paper/2504.05187/) (2025)
- [On the Efficiency and Robustness of Vibration-based Foundation Models for IoT Sensing: A Case Study](https://mlsanity.com/paper/2404.02461/) (2024)
- [Position Aware 60 GHz mmWave Beamforming for V2V Communications Utilizing Deep Learning](https://mlsanity.com/paper/2402.01259/) (2024)
- [Multi-Modality Sensing in mmWave Beamforming for Connected Vehicles Using Deep Learning](https://mlsanity.com/paper/2504.06173/) (2025)
- [Multi-Modal Data-Enhanced Foundation Models for Prediction and Control in Wireless Networks: A Survey](https://mlsanity.com/paper/2601.03181/) (2026)
- [Multi-modal Image and Radio Frequency Fusion for Optimizing Vehicle Positioning](https://mlsanity.com/paper/2410.19788/) (2024)
- [Multi-modal Data based Semi-Supervised Learning for Vehicle Positioning](https://mlsanity.com/paper/2410.20680/) (2024)
- [Large Model Enabled Embodied Intelligence for 6G Integrated Perception, Communication, and Computation Network](https://mlsanity.com/paper/2512.15109/) (2025)
- [Multi-Modal Sensing Aided mmWave Beamforming for V2V Communications with Transformers](https://mlsanity.com/paper/2509.11112/) (2025)
- [Equivariant Multi-agent Reinforcement Learning for Multimodal Vehicle-to-Infrastructure Systems](https://mlsanity.com/paper/2604.06914/) (2026)
- [Multi-Modal Sensing and Fusion in mmWave Beamforming for Connected Vehicles: A Transformer Based Framework](https://mlsanity.com/paper/2602.13606/) (2026)
- [From Passive Mirrors to Active Agents: Holonic Digital Twins for Physical AI over Networks](https://mlsanity.com/paper/2608.06227/) (2026)

## Abstract (arXiv)

Beam selection for millimeter-wave links in a vehicular scenario is a challenging problem, as an exhaustive search among all candidate beam pairs cannot be assuredly completed within short contact times. We solve this problem via a novel expediting beam selection by leveraging multimodal data collected from sensors like LiDAR, camera images, and GPS. We propose individual modality and distributed fusion-based deep learning (F-DL) architectures that can execute locally as well as at a mobile edge computing center (MEC), with a study on associated tradeoffs. We also formulate and solve an optimization problem that considers practical beam-searching, MEC processing and sensor-to-MEC data delivery latency overheads for determining the output dimensions of the above F-DL architectures. Results from extensive evaluations conducted on publicly available synthetic and home-grown real-world datasets reveal 95% and 96% improvement in beam selection speed over classical RF-only beam sweeping, respectively. F-DL also outperforms the state-of-the-art techniques by 20-22% in predicting top-10 best beam pairs.