# Robust Navigation with Cross-Modal Fusion and Knowledge Transfer

arXiv: https://arxiv.org/abs/2309.13266 · HTML: https://mlsanity.com/paper/2309.13266/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal distillation, one of 12 directions in Knowledge distillation.
- Builds on 27 of its 34 references that are in the map.
- First posted on arXiv 23 Sep 2023.

## Placed in

- Knowledge distillation / [Multimodal distillation](https://mlsanity.com/distill/multimodal-distillation/)

## Builds on (most cited first, 27 of 27)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels](https://mlsanity.com/paper/2004.13649/) (2020)
- [Object Goal Navigation using Goal-Oriented Semantic Exploration](https://mlsanity.com/paper/2007.00643/) (2020)
- [Reinforcement Learning with Unsupervised Auxiliary Tasks](https://mlsanity.com/paper/1611.05397/) (2016)
- [DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames](https://mlsanity.com/paper/1911.00357/) (2019)
- [Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges](https://mlsanity.com/paper/1902.07830/) (2019)
- [History Aware Multimodal Transformer for Vision-and-Language Navigation](https://mlsanity.com/paper/2110.13309/) (2021)
- [Deep Continuous Fusion for Multi-Sensor 3D Object Detection](https://mlsanity.com/paper/2012.10992/) (2020)
- [Learning to Navigate in Complex Environments](https://mlsanity.com/paper/1611.03673/) (2016)
- [Generalization in Reinforcement Learning by Soft Data Augmentation](https://mlsanity.com/paper/2011.13389/) (2020)
- [Sim-to-Real Reinforcement Learning for Deformable Object Manipulation](https://mlsanity.com/paper/1806.07851/) (2018)
- [Structured Scene Memory for Vision-Language Navigation](https://mlsanity.com/paper/2103.03454/) (2021)
- [Sim-to-Real Transfer for Vision-and-Language Navigation](https://mlsanity.com/paper/2011.03807/) (2020)
- [Self-Supervised Model Adaptation for Multimodal Semantic Segmentation](https://mlsanity.com/paper/1808.03833/) (2018)
- [Learning Navigation Behaviors End-to-End with AutoRL](https://mlsanity.com/paper/1809.10124/) (2018)
- [PRM-RL: Long-range Robotic Navigation Tasks by Combining Reinforcement Learning and Sampling-based Planning](https://mlsanity.com/paper/1710.03937/) (2017)
- [Meta Reinforcement Learning for Sim-to-real Domain Adaptation](https://mlsanity.com/paper/1909.12906/) (2019)
- [Distant Vehicle Detection Using Radar and Vision](https://mlsanity.com/paper/1901.10951/) (2019)
- [Long-Range Indoor Navigation with PRM-RL](https://mlsanity.com/paper/1902.09458/) (2019)
- [Integrating Egocentric Localization for More Realistic Point-Goal Navigation Agents](https://mlsanity.com/paper/2009.03231/) (2020)
- [The Surprising Effectiveness of Visual Odometry Techniques for Embodied PointGoal Navigation](https://mlsanity.com/paper/2108.11550/) (2021)
- [Continual Reinforcement Learning deployed in Real-life using Policy Distillation and Sim2Real Transfer](https://mlsanity.com/paper/1906.04452/) (2019)
- [Vision-based Navigation Using Deep Reinforcement Learning](https://mlsanity.com/paper/1908.03627/) (2019)
- [Domain Adaptation Through Task Distillation](https://mlsanity.com/paper/2008.11911/) (2020)

## Built on by (most cited first, 2 of 2)

- [Autonomous Navigation in Complex Environments](https://mlsanity.com/paper/2401.03267/) (2024)
- [NavTrust: Benchmarking Trustworthiness for Embodied Navigation](https://mlsanity.com/paper/2603.19229/) (2026)

## Abstract (arXiv)

Recently, learning-based approaches show promising results in navigation tasks. However, the poor generalization capability and the simulation-reality gap prevent a wide range of applications. We consider the problem of improving the generalization of mobile robots and achieving sim-to-real transfer for navigation skills. To that end, we propose a cross-modal fusion method and a knowledge transfer framework for better generalization. This is realized by a teacher-student distillation architecture. The teacher learns a discriminative representation and the near-perfect policy in an ideal environment. By imitating the behavior and representation of the teacher, the student is able to align the features from noisy multi-modal input and reduce the influence of variations on navigation policy. We evaluate our method in simulated and real-world environments. Experiments show that our method outperforms the baselines by a large margin and achieves robust navigation performance with varying working conditions.