# Grounded Vision-Language Navigation for UAVs with Open-Vocabulary Goal Understanding

arXiv: https://arxiv.org/abs/2506.10756 · HTML: https://mlsanity.com/paper/2506.10756/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 38 references that are in the map.
- First posted on arXiv 12 Jun 2025.

## Builds on (most cited first, 27 of 27)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015)
- [Object Goal Navigation using Goal-Oriented Semantic Exploration](https://mlsanity.com/paper/2007.00643/) (2020)
- [Gaussian Splatting SLAM](https://mlsanity.com/paper/2312.06741/) (2023)
- [NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigation](https://mlsanity.com/paper/2402.15852/) (2024)
- [ViNT: A Foundation Model for Visual Navigation](https://mlsanity.com/paper/2306.14846/) (2023)
- [VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigation](https://mlsanity.com/paper/2312.03275/) (2023)
- [Learning High-Speed Flight in the Wild](https://mlsanity.com/paper/2110.05113/) (2021)
- [AerialVLN: Vision-and-Language Navigation for UAVs](https://mlsanity.com/paper/2308.06735/) (2023)
- [Navigating to Objects in the Real World](https://mlsanity.com/paper/2212.00922/) (2022)
- [MapNav: A Novel Memory Representation via Annotated Semantic Maps for Vision-and-Language Navigation](https://mlsanity.com/paper/2502.13451/) (2025)
- [VoroNav: Voronoi-based Zero-shot Object Navigation with Large Language Model](https://mlsanity.com/paper/2401.02695/) (2024)
- [ViPlanner: Visual Semantic Imperative Learning for Local Navigation](https://mlsanity.com/paper/2310.00982/) (2023)
- [CityNav: A Large-Scale Dataset for Real-World Aerial Navigation](https://mlsanity.com/paper/2406.14240/) (2024)
- [CityBench: Evaluating the Capabilities of Large Language Models for Urban Tasks](https://mlsanity.com/paper/2406.13945/) (2024)
- [FLAME: Learning to Navigate with Multimodal LLM in Urban Environments](https://mlsanity.com/paper/2408.11051/) (2024)
- [Image-Goal Navigation in Complex Environments via Modular Learning](https://mlsanity.com/paper/2108.06105/) (2021)
- [Collaborative Target Search with a Visual Drone Swarm: An Adaptive Curriculum Embedded Multistage Reinforcement Learning Approach](https://mlsanity.com/paper/2204.12181/) (2022)
- [Learning Multi-Pursuit Evasion for Safe Targeted Navigation of Drones](https://mlsanity.com/paper/2304.03443/) (2023)
- [TC-SfM: Robust Track-Community-Based Structure-from-Motion](https://mlsanity.com/paper/2206.05866/) (2022)
- [Open-Architecture End-to-End System for Real-World Autonomous Robot Navigation](https://mlsanity.com/paper/2410.06239/) (2024)
- [VL-Nav: Neuro-Symbolic Reasoning-based Vision-Language Navigation](https://mlsanity.com/paper/2502.00931/) (2025)

## Built on by (most cited first, 19 of 19)

- [LongFly: Long-Horizon UAV Vision-and-Language Navigation with Spatiotemporal Context Integration](https://mlsanity.com/paper/2512.22010/) (2025)
- [π, But Make It Fly: Physics-Guided Transfer of VLA Models to Aerial Manipulation](https://mlsanity.com/paper/2603.25038/) (2026)
- [SINGER: An Onboard Generalist Vision-Language Navigation Policy for Drones](https://mlsanity.com/paper/2509.18610/) (2025)
- [APEX: A Decoupled Memory-based Explorer for Asynchronous Aerial Object Goal Navigation](https://mlsanity.com/paper/2602.00551/) (2026)
- [Fly0: Persistent Metric Anchoring for Zero-Shot Aerial Vision-Language Navigation](https://mlsanity.com/paper/2602.15875/) (2026)
- [Large Language Model-Assisted UAV Operations and Communications: A Multifaceted Survey and Tutorial](https://mlsanity.com/paper/2602.19534/) (2026)
- [FineCog-Nav: Integrating Fine-grained Cognitive Modules for Zero-shot Multimodal UAV Navigation](https://mlsanity.com/paper/2604.16298/) (2026)
- [ESARBench: A Benchmark for Agentic UAV Embodied Search and Rescue](https://mlsanity.com/paper/2605.01371/) (2026)
- [Embodied4C: Measuring What Matters for Embodied Vision-Language Navigation](https://mlsanity.com/paper/2512.18028/) (2025)
- [FSD-VLN: Fast-Slow Dual-System Modeling for Aerial Long-Horizon Vision-Language Navigation](https://mlsanity.com/paper/2607.08359/) (2026)
- [See-and-Reach: Precise Vision-Language Navigation for UAVs within the Field of View](https://mlsanity.com/paper/2606.20045/) (2026)
- [Uncertainty-Aware World Model for Aerial Image-Goal Navigation](https://mlsanity.com/paper/2608.05597/) (2026)
- [HumanDiffusion: A Vision-Based Diffusion Trajectory Planner with Human-Conditioned Goals for Search and Rescue UAV](https://mlsanity.com/paper/2601.14973/) (2026)
- [GaussFly: Contrastive Reinforcement Learning for Visuomotor Policies in 3D Gaussian Fields](https://mlsanity.com/paper/2604.05062/) (2026)
- [No Training, Better Flights: Test-Time Scaled VLMs for UAV Navigation](https://mlsanity.com/paper/2607.19288/) (2026)
- [RACO: Reliability-Aware Coarse-Goal Optimization for Inspection-Oriented UAV Vision-Language Navigation](https://mlsanity.com/paper/2608.22678/) (2026)
- [Map the Possibilities: Spatial Belief Fields for Language-Goal Aerial Navigation](https://mlsanity.com/paper/2609.05841/) (2026)
- [Dual-Layer Semantic-Spatial Belief Mapping for Aerial Object Goal Navigation](https://mlsanity.com/paper/2609.08164/) (2026)
- [Skytopia: Monocular Drone Navigation with Action-Conditioned Latent World Models](https://mlsanity.com/paper/2609.26007/) (2026)

## Abstract (arXiv)

Vision-and-language navigation (VLN) is a long-standing challenge in autonomous robotics, aiming to empower agents with the ability to follow human instructions while navigating complex environments. Two key bottlenecks remain in this field: generalization to out-of-distribution environments and reliance on fixed discrete action spaces. To address these challenges, we propose Vision-Language Fly (VLFly), a framework tailored for Unmanned Aerial Vehicles (UAVs) to execute language-guided flight. Without the requirement for localization or active ranging sensors, VLFly outputs continuous velocity commands purely from egocentric observations captured by an onboard monocular camera. The VLFly integrates three modules: an instruction encoder based on a large language model (LLM) that reformulates high-level language into structured prompts, a goal retriever powered by a vision-language model (VLM) that matches these prompts to goal images via vision-language similarity, and a waypoint planner that generates executable trajectories for real-time UAV control. VLFly is evaluated across diverse simulation environments without additional fine-tuning and consistently outperforms all baselines. Moreover, real-world VLN tasks in indoor and outdoor environments under direct and indirect instructions demonstrate that VLFly achieves robust open-vocabulary goal understanding and generalized navigation capabilities, even in the presence of abstract language input.