# VMLoc: Variational Fusion For Learning-Based Multimodal Camera Localization

arXiv: https://arxiv.org/abs/2003.07289 · HTML: https://mlsanity.com/paper/2003.07289/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Mar 2020.

## Built on by (most cited first, 13 of 13)

- [DynPoint: Dynamic Neural Point For View Synthesis](https://mlsanity.com/paper/2310.18999/) (2023)
- [Graph-based Thermal-Inertial SLAM with Probabilistic Neural Networks](https://mlsanity.com/paper/2104.07196/) (2021)
- [Learning Sequential Latent Variable Models from Multimodal Time Series Data](https://mlsanity.com/paper/2204.10419/) (2022)
- [Neural Surface Reconstruction from Sparse Views Using Epipolar Geometry](https://mlsanity.com/paper/2406.04301/) (2024)
- [DistillMatch: Leveraging Knowledge Distillation from Vision Foundation Model for Multimodal Image Matching](https://mlsanity.com/paper/2509.16017/) (2025)
- [FusionLoc: Camera-2D LiDAR Fusion Using Multi-Head Self-Attention for End-to-End Serving Robot Relocalization](https://mlsanity.com/paper/2303.06872/) (2023)
- [Sense, Imagine, Act: Multimodal Perception Improves Model-Based Reinforcement Learning for Head-to-Head Autonomous Racing](https://mlsanity.com/paper/2305.04750/) (2023)
- [WSCLoc: Weakly-Supervised Sparse-View Camera Relocalization](https://mlsanity.com/paper/2403.15272/) (2024)
- [Synergy-CLIP: Extending CLIP with Multi-modal Integration for Robust Representation Learning](https://mlsanity.com/paper/2504.21375/) (2025)
- [AnyMatch: Supercharging Universal Multi-Modal Image Matching with Large-Scale Single-View Images](https://mlsanity.com/paper/2606.31077/) (2026)
- [MambaLoc: Efficient Camera Localisation via State Space Model](https://mlsanity.com/paper/2408.09680/) (2024)
- [REMOTE: Real-time Ego-motion Tracking for Various Endoscopes via Multimodal Visual Feature Learning](https://mlsanity.com/paper/2501.18124/) (2025)
- [Bayesian Inverse Games with High-Dimensional Multi-Modal Observations](https://mlsanity.com/paper/2601.00696/) (2026)

## Abstract (arXiv)

Recent learning-based approaches have achieved impressive results in the field of single-shot camera localization. However, how best to fuse multiple modalities (e.g., image and depth) and to deal with degraded or missing input are less well studied. In particular, we note that previous approaches towards deep fusion do not perform significantly better than models employing a single modality. We conjecture that this is because of the naive approaches to feature space fusion through summation or concatenation which do not take into account the different strengths of each modality. To address this, we propose an end-to-end framework, termed VMLoc, to fuse different sensor inputs into a common latent space through a variational Product-of-Experts (PoE) followed by attention-based fusion. Unlike previous multimodal variational works directly adapting the objective function of vanilla variational auto-encoder, we show how camera localization can be accurately estimated through an unbiased objective function based on importance weighting. Our model is extensively evaluated on RGB-D datasets and the results prove the efficacy of our model. The source code is available at https://github.com/kaichen-z/VMLoc.