# Cross-Modal Reinforcement Learning for Navigation with Degraded Depth Measurements

arXiv: https://arxiv.org/abs/2603.22182 · HTML: https://mlsanity.com/paper/2603.22182/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for robot navigation and UAV control, one of 15 directions in Deep reinforcement learning methods.
- Builds on 23 of its 35 references that are in the map.
- First posted on arXiv 23 Mar 2026.

## Placed in

- Deep reinforcement learning methods / [RL for robot navigation and UAV control](https://mlsanity.com/deeprl/rl-for-robot-navigation-and-uav-control/)

## Builds on (most cited first, 23 of 23)

- [TartanAir: A Dataset to Push the Limits of Visual SLAM](https://mlsanity.com/paper/2003.14338/) (2020)
- [Wasserstein Auto-Encoders](https://mlsanity.com/paper/1711.01558/) (2017)
- [Multimodal Deep Learning](https://mlsanity.com/paper/2301.04856/) (2023)
- [Cross Modal Distillation for Supervision Transfer](https://mlsanity.com/paper/1507.00448/) (2015)
- [Sparse-to-Dense: Depth Prediction from Sparse Depth Samples and a Single Image](https://mlsanity.com/paper/1709.07492/) (2017)
- [Learning High-Speed Flight in the Wild](https://mlsanity.com/paper/2110.05113/) (2021)
- [Sample Factory: Egocentric 3D Control from Pixels at 100000 FPS with Asynchronous Reinforcement Learning](https://mlsanity.com/paper/2006.11751/) (2020)
- [Perception and Sensing for Autonomous Vehicles Under Adverse Weather Conditions: A Survey](https://mlsanity.com/paper/2112.08936/) (2021)
- [Uncertainty Quantification with Statistical Guarantees in End-to-End Autonomous Driving Control](https://mlsanity.com/paper/1909.09884/) (2019)
- [Learning to Fly by Crashing](https://mlsanity.com/paper/1704.05588/) (2017)
- [Learning Perception-Aware Agile Flight in Cluttered Environments](https://mlsanity.com/paper/2210.01841/) (2022)
- [CERBERUS: Autonomous Legged and Aerial Robotic Exploration in the Tunnel and Urban Circuits of the DARPA Subterranean Challenge](https://mlsanity.com/paper/2201.07067/) (2022)
- [Aerial Gym Simulator: A Framework for Highly Parallelized Simulation of Aerial Robots](https://mlsanity.com/paper/2503.01471/) (2025)
- [Reinforcement Learning for Collision-free Flight Exploiting Deep Collision Encoding](https://mlsanity.com/paper/2402.03947/) (2024)
- [Learning an Efficient Multimodal Depth Completion Model](https://mlsanity.com/paper/2208.10771/) (2022)
- [Multi-level Cross-modal Interaction Network for RGB-D Salient Object Detection](https://mlsanity.com/paper/2007.14352/) (2020)
- [Stacked Wasserstein Autoencoder](https://mlsanity.com/paper/1910.02560/) (2019)
- [Vision-Based Deep Reinforcement Learning of UAV Autonomous Navigation Using Privileged Information](https://mlsanity.com/paper/2412.06313/) (2024)
- [Degradation Resilient LiDAR-Radar-Inertial Odometry](https://mlsanity.com/paper/2403.05332/) (2024)
- [Robust High-Speed State Estimation for Off-road Navigation using Radar Velocity Factors](https://mlsanity.com/paper/2409.10953/) (2024)
- [Cross-modal feature fusion for robust point cloud registration with ambiguous geometry](https://mlsanity.com/paper/2505.13088/) (2025)
- [Vision-Depth Landmarks and Inertial Fusion for Navigation in Degraded Visual Environments](https://mlsanity.com/paper/1903.01659/) (2019)
- [Enhancing Deep Reinforcement Learning-based Robot Navigation Generalization through Scenario Augmentation](https://mlsanity.com/paper/2503.01146/) (2025)

## Abstract (arXiv)

This paper presents a cross-modal learning framework that exploits complementary information from depth and grayscale images for robust navigation. We introduce a Cross-Modal Wasserstein Autoencoder that learns shared latent representations by enforcing cross-modal consistency, enabling the system to infer depth-relevant features from grayscale observations when depth measurements are corrupted. The learned representations are integrated with a Reinforcement Learning-based policy for collision-free navigation in unstructured environments when depth sensors experience degradation due to adverse conditions such as poor lighting or reflective surfaces. Simulation and real-world experiments demonstrate that our approach maintains robust performance under significant depth degradation and successfully transfers to real environments.