# Variational End-to-End Navigation and Localization

arXiv: https://arxiv.org/abs/1811.10119 · HTML: https://mlsanity.com/paper/1811.10119/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Nov 2018.

## Built on by (most cited first, 30 of 34)

- [Urban Driving with Conditional Imitation Learning](https://mlsanity.com/paper/1912.00177/) (2019)
- [VISTA 2.0: An Open, Data-driven Simulator for Multimodal Sensing and Policy Learning for Autonomous Vehicles](https://mlsanity.com/paper/2111.12083/) (2021)
- [Drive Anywhere: Generalizable End-to-end Autonomous Driving with Multi-modal Foundation Models](https://mlsanity.com/paper/2310.17642/) (2023)
- [Deep Federated Learning for Autonomous Driving](https://mlsanity.com/paper/2110.05754/) (2021)
- [Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications](https://mlsanity.com/paper/2204.11786/) (2022)
- [Safe Real-World Autonomous Driving by Learning to Predict and Plan with a Mixture of Experts](https://mlsanity.com/paper/2211.02131/) (2022)
- [Efficient and Robust LiDAR-Based End-to-End Navigation](https://mlsanity.com/paper/2105.09932/) (2021)
- [Learning Interactive Driving Policies via Data-driven Simulation](https://mlsanity.com/paper/2111.12137/) (2021)
- [Trajectory Prediction with Linguistic Representations](https://mlsanity.com/paper/2110.09741/) (2021)
- [Enhancing Online Road Network Perception and Reasoning with Standard Definition Maps](https://mlsanity.com/paper/2408.01471/) (2024)
- [Differentiable Control Barrier Functions for Vision-based End-to-End Autonomous Driving](https://mlsanity.com/paper/2203.02401/) (2022)
- [Imitation Learning of Hierarchical Driving Model: from Continuous Intention to Continuous Trajectory](https://mlsanity.com/paper/2010.10393/) (2020)
- [PathRL: An End-to-End Path Generation Method for Collision Avoidance via Deep Reinforcement Learning](https://mlsanity.com/paper/2310.13295/) (2023)
- [End-to-end Autonomous Driving using Deep Learning: A Systematic Review](https://mlsanity.com/paper/2311.18636/) (2023)
- [Trajectory Prediction for Autonomous Driving with Topometric Map](https://mlsanity.com/paper/2105.03869/) (2021)
- [Deep Visual Navigation under Partial Observability](https://mlsanity.com/paper/2109.07752/) (2021)
- [Learn by Observation: Imitation Learning for Drone Patrolling from Videos of A Human Navigator](https://mlsanity.com/paper/2008.13193/) (2020)
- [Learning Stability Attention in Vision-based End-to-end Driving Policies](https://mlsanity.com/paper/2304.02733/) (2023)
- [Domain Generalization for Vision-based Driving Trajectory Generation](https://mlsanity.com/paper/2109.13858/) (2021)
- [See Less, Drive Better: Generalizable End-to-End Autonomous Driving via Foundation Models Stochastic Patch Selection](https://mlsanity.com/paper/2601.10707/) (2026)
- [Learning to Predict Navigational Patterns from Partial Observations](https://mlsanity.com/paper/2304.13242/) (2023)
- [Self-Supervised Interpretable End-to-End Learning via Latent Functional Modularity](https://mlsanity.com/paper/2403.18947/) (2024)
- [Formal Verification of Stochastic Systems with ReLU Neural Network Controllers](https://mlsanity.com/paper/2103.05142/) (2021)
- [Learning a Model for Inferring a Spatial Road Lane Network Graph using Self-Supervision](https://mlsanity.com/paper/2107.01784/) (2021)
- [On the Choice of Data for Efficient Training and Validation of End-to-End Driving Models](https://mlsanity.com/paper/2206.00608/) (2022)
- [Learning autonomous driving from aerial imagery](https://mlsanity.com/paper/2410.14177/) (2024)
- [SSF-PAN: Semantic Scene Flow-Based Perception for Autonomous Navigation in Traffic Scenarios](https://mlsanity.com/paper/2501.16754/) (2025)
- [Robustness Is a Function, Not a Number: A Factorized Comprehensive Study of OOD Robustness in Vision-Based Driving](https://mlsanity.com/paper/2602.09018/) (2026)
- [Perception Helps Planning: Facilitating Multi-Stage Lane-Level Integration via Double-Edge Structures](https://mlsanity.com/paper/2407.11644/) (2024)
- [Inside Knowledge: Graph-based Path Generation with Explainable Data Augmentation and Curriculum Learning for Visual Indoor Navigation](https://mlsanity.com/paper/2508.11446/) (2025)

## Abstract (arXiv)

Deep learning has revolutionized the ability to learn "end-to-end" autonomous vehicle control directly from raw sensory data. While there have been recent extensions to handle forms of navigation instruction, these works are unable to capture the full distribution of possible actions that could be taken and to reason about localization of the robot within the environment. In this paper, we extend end-to-end driving networks with the ability to perform point-to-point navigation as well as probabilistic localization using only noisy GPS data. We define a novel variational network capable of learning from raw camera data of the environment as well as higher level roadmaps to predict (1) a full probability distribution over the possible control commands; and (2) a deterministic control command capable of navigating on the route specified within the map. Additionally, we formulate how our model can be used to localize the robot according to correspondences between the map and the observed visual road topology, inspired by the rough localization that human drivers can perform. We test our algorithms on real-world driving data that the vehicle has never driven through before, and integrate our point-to-point navigation algorithms onboard a full-scale autonomous vehicle for real-time performance. Our localization algorithm is also evaluated over a new set of roads and intersections to demonstrates rough pose localization even in situations without any GPS prior.