# Multi-modal Image and Radio Frequency Fusion for Optimizing Vehicle Positioning

arXiv: https://arxiv.org/abs/2410.19788 · HTML: https://mlsanity.com/paper/2410.19788/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 41 references that are in the map.
- First posted on arXiv 15 Oct 2024.

## Builds on (most cited first, 21 of 21)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Learning to Reweight Examples for Robust Deep Learning](https://mlsanity.com/paper/1803.09050/) (2018)
- [A Joint Learning and Communications Framework for Federated Learning over Wireless Networks](https://mlsanity.com/paper/1909.07972/) (2019)
- [Distributed Learning in Wireless Networks: Recent Progress and Future Challenges](https://mlsanity.com/paper/2104.02151/) (2021)
- [Convolutional Neural Networks at Constrained Time Cost](https://mlsanity.com/paper/1412.1710/) (2014)
- [Deep Bilevel Learning](https://mlsanity.com/paper/1809.01465/) (2018)
- [Vision-Aided 6G Wireless Communications: Blockage Prediction and Proactive Handoff](https://mlsanity.com/paper/2102.09527/) (2021)
- [Millimeter Wave Base Stations with Cameras: Vision Aided Beam and Blockage Prediction](https://mlsanity.com/paper/1911.06255/) (2019)
- [Deep Learning on Multimodal Sensor Data at the Wireless Edge for Vehicular Network](https://mlsanity.com/paper/2201.04712/) (2022)
- [Computer Vision Aided Beam Tracking in A Real-World Millimeter Wave Deployment](https://mlsanity.com/paper/2111.14803/) (2021)
- [ViWi Vision-Aided mmWave Beam Tracking: Dataset, Task, and Baseline Solutions](https://mlsanity.com/paper/2002.02445/) (2020)
- [CSI-Based Multi-Antenna and Multi-Point Indoor Positioning Using Probability Fusion](https://mlsanity.com/paper/2009.02798/) (2020)
- [Computer Vision Aided Blockage Prediction in Real-World Millimeter Wave Deployments](https://mlsanity.com/paper/2203.01907/) (2022)
- [CSI-Based Localization with CNNs Exploiting Phase Information](https://mlsanity.com/paper/2101.08983/) (2021)
- [CSI-based Outdoor Localization for Massive MIMO: Experiments with a Learning Approach](https://mlsanity.com/paper/1806.07447/) (2018)
- [Multi-Environment based Meta-Learning with CSI Fingerprints for Radio Based Positioning](https://mlsanity.com/paper/2210.14510/) (2022)
- [Complex-Valued Neural Network based Federated Learning for Multi-user Indoor Positioning Performance Optimization](https://mlsanity.com/paper/2403.00665/) (2024)
- [Positioning Using Visible Light Communications: A Perspective Arcs Approach](https://mlsanity.com/paper/2204.08131/) (2022)
- [Unsupervised Person Re-Identification with Wireless Positioning under Weak Scene Labeling](https://mlsanity.com/paper/2110.15610/) (2021)
- [Automated Static Camera Calibration with Intelligent Vehicles](https://mlsanity.com/paper/2304.10814/) (2023)

## Built on by (most cited first, 2 of 2)

- [Equivariant Multi-agent Reinforcement Learning for Multimodal Vehicle-to-Infrastructure Systems](https://mlsanity.com/paper/2604.06914/) (2026)
- [Vision-Wireless Fusion for Multi-User Localization: A Cross-Modal Transformer Approach](https://mlsanity.com/paper/2609.23372/) (2026)

## Abstract (arXiv)

In this paper, a multi-modal vehicle positioning framework that jointly localizes vehicles with channel state information (CSI) and images is designed. In particular, we consider an outdoor scenario where each vehicle can communicate with only one BS, and hence, it can upload its estimated CSI to only its associated BS. Each BS is equipped with a set of cameras, such that it can collect a small number of labeled CSI, a large number of unlabeled CSI, and the images taken by cameras. To exploit the unlabeled CSI data and position labels obtained from images, we design an meta-learning based hard expectation-maximization (EM) algorithm. Specifically, since we do not know the corresponding relationship between unlabeled CSI and the multiple vehicle locations in images, we formulate the calculation of the training objective as a minimum matching problem. To reduce the impact of label noises caused by incorrect matching between unlabeled CSI and vehicle locations obtained from images and achieve better convergence, we introduce a weighted loss function on the unlabeled datasets, and study the use of a meta-learning algorithm for computing the weighted loss. Subsequently, the model parameters are updated according to the weighted loss function of unlabeled CSI samples and their matched position labels obtained from images. Simulation results show that the proposed method can reduce the positioning error by up to 61% compared to a baseline that does not use images and uses only CSI fingerprint for vehicle positioning.