# Learning End-to-end Multimodal Sensor Policies for Autonomous Navigation

arXiv: https://arxiv.org/abs/1705.10422 · HTML: https://mlsanity.com/paper/1705.10422/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 May 2017.

## Built on by (most cited first, 18 of 18)

- [Reward (Mis)design for Autonomous Driving](https://mlsanity.com/paper/2104.13906/) (2021)
- [Embodied Foundation Models at the Edge: A Survey of Deployment Constraints and Mitigation Strategies](https://mlsanity.com/paper/2603.16952/) (2026)
- [Touch-based Curiosity for Sparse-Reward Tasks](https://mlsanity.com/paper/2104.00442/) (2021)
- [Masked Imitation Learning: Discovering Environment-Invariant Modalities in Multimodal Demonstrations](https://mlsanity.com/paper/2209.07682/) (2022)
- [TraVLR: Now You See It, Now You Don't! A Bimodal Dataset for Evaluating Visio-Linguistic Reasoning](https://mlsanity.com/paper/2111.10756/) (2021)
- [World Model Robustness via Surprise Recognition](https://mlsanity.com/paper/2512.01119/) (2025)
- [Investigating the Effect of Sensor Modalities in Multi-Sensor Detection-Prediction Models](https://mlsanity.com/paper/2101.03279/) (2021)
- [Self-Supervised Multisensory Pretraining for Contact-Rich Robot Reinforcement Learning](https://mlsanity.com/paper/2511.14427/) (2025)
- [Deep Learning for Embodied Vision Navigation: A Survey](https://mlsanity.com/paper/2108.04097/) (2021)
- [BronchoCopilot: Towards Autonomous Robotic Bronchoscopy via Multimodal Reinforcement Learning](https://mlsanity.com/paper/2403.01483/) (2024)
- [Inductive Learning of Robot Task Knowledge from Raw Data and Online Expert Feedback](https://mlsanity.com/paper/2501.07507/) (2025)
- [Effective Multimodal Reinforcement Learning with Modality Alignment and Importance Enhancement](https://mlsanity.com/paper/2302.09318/) (2023)
- [Stackelberg Coupling of Online Representation Learning and Reinforcement Learning](https://mlsanity.com/paper/2508.07452/) (2025)
- [RAPID: Reconfigurable, Adaptive Platform for Iterative Design](https://mlsanity.com/paper/2602.06653/) (2026)
- [FusionSense: Tri-Stage Near-Sensor Learning for Runtime-Adaptive Multimodal Edge Intelligence](https://mlsanity.com/paper/2605.22868/) (2026)
- [ShapKO: Shapley-Adaptive Modality Knockout for Robust Multimodal Learning](https://mlsanity.com/paper/2607.09884/) (2026)
- [Sensing Which Modality Matters: Evidence-Gated Regularization for Robust VLA Policies](https://mlsanity.com/paper/2609.03142/) (2026)
- [SpikeOcc: Towards Resilient Multimodal 3D Occupancy Perception via Spiking Experts](https://mlsanity.com/paper/2610.15745/) (2026)

## Abstract (arXiv)

Multisensory polices are known to enhance both state estimation and target tracking. However, in the space of end-to-end sensorimotor control, this multi-sensor outlook has received limited attention. Moreover, systematic ways to make policies robust to partial sensor failure are not well explored. In this work, we propose a specific customization of Dropout, called textitSensor Dropout, to improve multisensory policy robustness and handle partial failure in the sensor-set. We also introduce an additional auxiliary loss on the policy network in order to reduce variance in the band of potential multi- and uni-sensory policies to reduce jerks during policy switching triggered by an abrupt sensor failure or deactivation/activation. Finally, through the visualization of gradients, we show that the learned policies are conditioned on the same latent states representation despite having diverse observations spaces - a hallmark of true sensor-fusion. Simulation results of the multisensory policy, as visualized in TORCS racing game, can be seen here: https://youtu.be/QAK2lcXjNZc.