# Learning Visuomotor Policies for Aerial Navigation Using Cross-Modal Representations

arXiv: https://arxiv.org/abs/1909.06993 · HTML: https://mlsanity.com/paper/1909.06993/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Sep 2019.

## Built on by (most cited first, 19 of 19)

- [DeepSMOTE: Fusing Deep Learning and SMOTE for Imbalanced Data](https://mlsanity.com/paper/2105.02340/) (2021)
- [Machine Learning-Aided Operations and Communications of Unmanned Aerial Vehicles: A Contemporary Survey](https://mlsanity.com/paper/2211.04324/) (2022)
- [ContraNeRF: Generalizable Neural Radiance Fields for Synthetic-to-real Novel View Synthesis via Contrastive Learning](https://mlsanity.com/paper/2303.11052/) (2023)
- [Depth-CUPRL: Depth-Imaged Contrastive Unsupervised Prioritized Representations in Reinforcement Learning for Mapless Navigation of Unmanned Aerial Vehicles](https://mlsanity.com/paper/2206.15211/) (2022)
- [TANGO: Traversability-Aware Navigation with Local Metric Control for Topological Goals](https://mlsanity.com/paper/2509.08699/) (2025)
- [GRID: A Platform for General Robot Intelligence Development](https://mlsanity.com/paper/2310.00887/) (2023)
- [Increasing the Efficiency of Policy Learning for Autonomous Vehicles by Multi-Task Representation Learning](https://mlsanity.com/paper/2103.14718/) (2021)
- [ReaDy-Go: Real-to-Sim Dynamic 3D Gaussian Splatting Simulation for Environment-Specific Visual Navigation with Moving Obstacles](https://mlsanity.com/paper/2602.11575/) (2026)
- [Robust Deep Reinforcement Learning for Quadcopter Control](https://mlsanity.com/paper/2111.03915/) (2021)
- [BatMobility: Towards Flying Without Seeing for Autonomous Drones](https://mlsanity.com/paper/2307.11518/) (2023)
- [SpecGuard: Specification Aware Recovery for Robotic Autonomous Vehicles from Physical Attacks](https://mlsanity.com/paper/2408.15200/) (2024)
- [Cross-Modal Contrastive Learning of Representations for Navigation using Lightweight, Low-Cost Millimeter Wave Radar for Adverse Environmental Conditions](https://mlsanity.com/paper/2101.03525/) (2021)
- [An Empirical Evaluation of Four Off-the-Shelf Proprietary Visual-Inertial Odometry Systems](https://mlsanity.com/paper/2207.06780/) (2022)
- [Towards Dependable Autonomous Systems Based on Bayesian Deep Learning Components](https://mlsanity.com/paper/2301.05297/) (2023)
- [Tube-NeRF: Efficient Imitation Learning of Visuomotor Policies from MPC using Tube-Guided Data Augmentation and NeRFs](https://mlsanity.com/paper/2311.14153/) (2023)
- [Invisible Servoing: a Visual Servoing Approach with Return-Conditioned Latent Diffusion](https://mlsanity.com/paper/2409.13337/) (2024)
- [Oracle-Guided Masked Contrastive Reinforcement Learning for Visuomotor Policies](https://mlsanity.com/paper/2510.05692/) (2025)
- [Self-Predictive Representation for Autonomous UAV Object-Goal Navigation](https://mlsanity.com/paper/2604.21130/) (2026)
- [Distilling Privileged Control Barrier Functions into RGB-Only Safety Filters for Dynamic Visual Navigation](https://mlsanity.com/paper/2609.36520/) (2026)

## Abstract (arXiv)

Machines are a long way from robustly solving open-world perception-control tasks, such as first-person view (FPV) aerial navigation. While recent advances in end-to-end Machine Learning, especially Imitation and Reinforcement Learning appear promising, they are constrained by the need of large amounts of difficult-to-collect labeled real-world data. Simulated data, on the other hand, is easy to generate, but generally does not render safe behaviors in diverse real-life scenarios. In this work we propose a novel method for learning robust visuomotor policies for real-world deployment which can be trained purely with simulated data. We develop rich state representations that combine supervised and unsupervised environment data. Our approach takes a cross-modal perspective, where separate modalities correspond to the raw camera data and the system states relevant to the task, such as the relative pose of gates to the drone in the case of drone racing. We feed both data modalities into a novel factored architecture, which learns a joint low-dimensional embedding via Variational Auto Encoders. This compact representation is then fed into a control policy, which we trained using imitation learning with expert trajectories in a simulator. We analyze the rich latent spaces learned with our proposed representations, and show that the use of our cross-modal architecture significantly improves control policy performance as compared to end-to-end learning or purely unsupervised feature extractors. We also present real-world results for drone navigation through gates in different track configurations and environmental conditions. Our proposed method, which runs fully onboard, can successfully generalize the learned representations and policies across simulation and reality, significantly outperforming baseline approaches. Supplementary video: https://youtu.be/VKc3A5HlUU8