# Self-Supervised Learning to Fly using Efficient Semantic Segmentation and Metric Depth Estimation for Low-Cost Autonomous UAVs

arXiv: https://arxiv.org/abs/2510.16624 · HTML: https://mlsanity.com/paper/2510.16624/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 35 of its 69 references that are in the map.
- First posted on arXiv 18 Oct 2025.

## Builds on (most cited first, 30 of 35)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [YOLACT: Real-time Instance Segmentation](https://mlsanity.com/paper/1904.02689/) (2019)
- [Depth Pro: Sharp Monocular Metric Depth in Less Than a Second](https://mlsanity.com/paper/2410.02073/) (2024)
- [OneFormer: One Transformer to Rule Universal Image Segmentation](https://mlsanity.com/paper/2211.06220/) (2022)
- [MoGe: Unlocking Accurate Monocular Geometry Estimation for Open-Domain Images with Optimal Training Supervision](https://mlsanity.com/paper/2410.19115/) (2024)
- [MoGe-2: Accurate Monocular Geometry with Metric Scale and Sharp Details](https://mlsanity.com/paper/2507.02546/) (2025)
- [Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video](https://mlsanity.com/paper/1908.10553/) (2019)
- [Deep Drone Racing: From Simulation to Reality with Domain Randomization](https://mlsanity.com/paper/1905.09727/) (2019)
- [Unsupervised Scale-consistent Depth Learning from Video](https://mlsanity.com/paper/2105.11610/) (2021)
- [Autonomous Drone Racing with Deep Reinforcement Learning](https://mlsanity.com/paper/2103.08624/) (2021)
- [SC-DepthV3: Robust Self-supervised Monocular Depth Estimation for Dynamic Scenes](https://mlsanity.com/paper/2211.03660/) (2022)
- [Auto-Rectify Network for Unsupervised Indoor Depth Estimation](https://mlsanity.com/paper/2006.02708/) (2020)
- [AlphaPilot: Autonomous Drone Racing](https://mlsanity.com/paper/2005.12813/) (2020)
- [A 64mW DNN-based Visual Navigation Engine for Autonomous Nano-Drones](https://mlsanity.com/paper/1805.01831/) (2018)
- [A Real-Time Game Theoretic Planner for Autonomous Two-Player Drone Racing](https://mlsanity.com/paper/1801.02302/) (2018)
- [Toward Low-Flying Autonomous MAV Trail Navigation using Deep Neural Networks for Environmental Awareness](https://mlsanity.com/paper/1705.02550/) (2017)
- [Learning Deep Sensorimotor Policies for Vision-based Autonomous Drone Racing](https://mlsanity.com/paper/2210.14985/) (2022)
- [Vision Transformers for End-to-End Vision-Based Quadrotor Obstacle Avoidance](https://mlsanity.com/paper/2405.10391/) (2024)
- [Learning Vision-based Flight in Drone Swarms by Imitation](https://mlsanity.com/paper/1908.02999/) (2019)
- [A Safer Vision-based Autonomous Planning System for Quadrotor UAVs with Dynamic Obstacle Trajectory Prediction and Its Application with LLMs](https://mlsanity.com/paper/2311.12893/) (2023)
- [From Monocular SLAM to Autonomous Drone Exploration](https://mlsanity.com/paper/1609.07835/) (2016)
- [TransVisDrone: Spatio-Temporal Transformer for Vision-based Drone-to-Drone Detection in Aerial Videos](https://mlsanity.com/paper/2210.08423/) (2022)
- [Self-supervised Hypergraphs for Learning Multiple World Interpretations](https://mlsanity.com/paper/2308.07615/) (2023)
- [Autonomous drone race: A computationally efficient vision-based navigation and control strategy](https://mlsanity.com/paper/1809.05958/) (2018)
- [Autonomous Recharging and Flight Mission Planning for Battery-operated Autonomous Drones](https://mlsanity.com/paper/1703.10049/) (2017)
- [Towards Automatic Annotation for Semantic Segmentation in Drone Videos](https://mlsanity.com/paper/1910.10026/) (2019)
- [AutoTrans: A Complete Planning and Control Framework for Autonomous UAV Payload Transportation](https://mlsanity.com/paper/2310.15050/) (2023)

## Abstract (arXiv)

This paper presents a vision-only autonomous flight system for small UAVs operating in controlled indoor environments. The system combines semantic segmentation with monocular depth estimation to enable obstacle avoidance, scene exploration, and autonomous safe landing operations without requiring GPS or expensive sensors such as LiDAR. A key innovation is an adaptive scale factor algorithm that converts non-metric monocular depth predictions into accurate metric distance measurements by leveraging semantic ground plane detection and camera intrinsic parameters, achieving a mean distance error of 14.4 cm. The approach uses a knowledge distillation framework where a color-based Support Vector Machine (SVM) teacher generates training data for a lightweight U-Net student network (1.6M parameters) capable of real-time semantic segmentation. For more complex environments, the SVM teacher can be replaced with a state-of-the-art segmentation model. Testing was conducted in a controlled 5x4 meter laboratory environment with eight cardboard obstacles simulating urban structures. Extensive validation across 30 flight tests in a real-world environment and 100 flight tests in a digital-twin environment demonstrates that the combined segmentation and depth approach increases the distance traveled during surveillance and reduces mission time while maintaining 100% success rates. The system is further optimized through end-to-end learning, where a compact student neural network learns complete flight policies from demonstration data generated by our best-performing method, achieving an 87.5% autonomous mission success rate. This work advances practical vision-based drone navigation in structured environments, demonstrating solutions for metric depth estimation and computational efficiency challenges that enable deployment on resource-constrained platforms.