# An End-to-End Learning-Based Multi-Sensor Fusion for Autonomous Vehicle Localization

arXiv: https://arxiv.org/abs/2503.05088 · HTML: https://mlsanity.com/paper/2503.05088/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in End-to-end system architecture, one of 9 directions in End-to-end autonomous driving.
- Builds on 8 of its 25 references that are in the map.
- First posted on arXiv 7 Mar 2025.

## Placed in

- End-to-end autonomous driving / [End-to-end system architecture](https://mlsanity.com/driving/end-to-end-system-architecture/)

## Builds on (most cited first, 8 of 8)

- [KalmanNet: Neural Network Aided Kalman Filtering for Partially Known Dynamics](https://mlsanity.com/paper/2107.10043/) (2021)
- [TLIO: Tight Learned Inertial Odometry](https://mlsanity.com/paper/2007.01867/) (2020)
- [Backprop KF: Learning Discriminative Deterministic State Estimators](https://mlsanity.com/paper/1605.07148/) (2016)
- [AI-IMU Dead-Reckoning](https://mlsanity.com/paper/1904.06064/) (2019)
- [Multivariate Uncertainty in Deep Learning](https://mlsanity.com/paper/1910.14215/) (2019)
- [Adaptive Adjustment of Noise Covariance in Kalman Filter for Dynamic State Estimation](https://mlsanity.com/paper/1702.00884/) (2017)
- [CELLO-3D: Estimating the Covariance of ICP in the Real World](https://mlsanity.com/paper/1810.01470/) (2018)
- [Learning-based NLOS Detection and Uncertainty Prediction of GNSS Observations with Transformer-Enhanced LSTM Network](https://mlsanity.com/paper/2309.00480/) (2023)

## Abstract (arXiv)

Multi-sensor fusion is essential for autonomous vehicle localization, as it is capable of integrating data from various sources for enhanced accuracy and reliability. The accuracy of the integrated location and orientation depends on the precision of the uncertainty modeling. Traditional methods of uncertainty modeling typically assume a Gaussian distribution and involve manual heuristic parameter tuning. However, these methods struggle to scale effectively and address long-tail scenarios. To address these challenges, we propose a learning-based method that encodes sensor information using higher-order neural network features, thereby eliminating the need for uncertainty estimation. This method significantly eliminates the need for parameter fine-tuning by developing an end-to-end neural network that is specifically designed for multi-sensor fusion. In our experiments, we demonstrate the effectiveness of our approach in real-world autonomous driving scenarios. Results show that the proposed method outperforms existing multi-sensor fusion methods in terms of both accuracy and robustness. A video of the results can be viewed at https://youtu.be/q4iuobMbjME.