# VLM-RRT: Vision Language Model Guided RRT Search for Autonomous UAV Navigation

arXiv: https://arxiv.org/abs/2505.23267 · HTML: https://mlsanity.com/paper/2505.23267/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 25 of its 44 references that are in the map.
- First posted on arXiv 29 May 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 25 of 25)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models](https://mlsanity.com/paper/2305.16986/) (2023)
- [Informed RRT*: Optimal Sampling-based Path Planning Focused via Direct Sampling of an Admissible Ellipsoidal Heuristic](https://mlsanity.com/paper/1404.2334/) (2014)
- [LLM-A*: Large Language Model Enhanced Incremental Heuristic Search on Path Planning](https://mlsanity.com/paper/2407.02511/) (2024)
- [Deep sequential models for sampling-based planning](https://mlsanity.com/paper/1810.00804/) (2018)
- [Multi-UAV Coverage Path Planning for the Inspection of Large and Complex Structures](https://mlsanity.com/paper/2007.13065/) (2020)
- [Neural Informed RRT*: Learning-based Path Planning with Point Cloud State Representations under Admissible Ellipsoidal Constraints](https://mlsanity.com/paper/2309.14595/) (2023)
- [Synergising Human-like Responses and Machine Intelligence for Planning in Disaster Response](https://mlsanity.com/paper/2404.09877/) (2024)
- [Probabilistic Search and Track with Multiple Mobile Agents](https://mlsanity.com/paper/2302.00518/) (2023)
- [Towards Automated 3D Search Planning for Emergency Response Missions](https://mlsanity.com/paper/2304.03570/) (2023)
- [Jointly-optimized Trajectory Generation and Camera Control for 3D Coverage Planning](https://mlsanity.com/paper/2504.05887/) (2025)
- [3D Trajectory Planning for UAV-based Search Missions: An Integrated Assessment and Search Planning Approach](https://mlsanity.com/paper/2302.12587/) (2023)
- [Distributed Search Planning in 3-D Environments With a Dynamically Varying Number of Agents](https://mlsanity.com/paper/2304.08932/) (2023)
- [UAV-based Receding Horizon Control for 3D Inspection Planning](https://mlsanity.com/paper/2304.10201/) (2023)
- [Distributed Estimation and Control for Jamming an Aerial Target With Multiple Agents](https://mlsanity.com/paper/2305.13919/) (2023)
- [Joint Estimation and Control for Multi-Target Passive Monitoring with an Autonomous UAV Agent](https://mlsanity.com/paper/2307.06633/) (2023)
- [Cooperative Receding Horizon 3D Coverage Control with a Team of Networked Aerial Agents](https://mlsanity.com/paper/2401.15674/) (2024)
- [Hierarchical Fault-Tolerant Coverage Control for an Autonomous Aerial Agent](https://mlsanity.com/paper/2404.09838/) (2024)
- [Probabilistically Robust Trajectory Planning of Multiple Aerial Agents](https://mlsanity.com/paper/2409.12718/) (2024)
- [RRT-CBF Based Motion Planning](https://mlsanity.com/paper/2410.00343/) (2024)
- [Rolling Horizon Coverage Control with Collaborative Autonomous Agents](https://mlsanity.com/paper/2504.05883/) (2025)

## Built on by (most cited first, 5 of 5)

- [UAVs Meet Agentic AI: A Multidomain Survey of Autonomous Aerial Intelligence and Agentic UAVs](https://mlsanity.com/paper/2506.08045/) (2025)
- [Is your VLM Sky-Ready? A Comprehensive Spatial Intelligence Benchmark for UAV Navigation](https://mlsanity.com/paper/2511.13269/) (2025)
- [FineCog-Nav: Integrating Fine-grained Cognitive Modules for Zero-shot Multimodal UAV Navigation](https://mlsanity.com/paper/2604.16298/) (2026)
- [FlightDiffusion: Revolutionising Autonomous Drone Training with Diffusion Models Generating FPV Video](https://mlsanity.com/paper/2509.14082/) (2025)
- [MMP-A*: Multimodal Perception Enhanced Incremental Heuristic Search on Path Planning](https://mlsanity.com/paper/2601.01910/) (2026)

## Abstract (arXiv)

Path planning is a fundamental capability of autonomous Unmanned Aerial Vehicles (UAVs), enabling them to efficiently navigate toward a target region or explore complex environments while avoiding obstacles. Traditional pathplanning methods, such as Rapidly-exploring Random Trees (RRT), have proven effective but often encounter significant challenges. These include high search space complexity, suboptimal path quality, and slow convergence, issues that are particularly problematic in high-stakes applications like disaster response, where rapid and efficient planning is critical. To address these limitations and enhance path-planning efficiency, we propose Vision Language Model RRT (VLM-RRT), a hybrid approach that integrates the pattern recognition capabilities of Vision Language Models (VLMs) with the path-planning strengths of RRT. By leveraging VLMs to provide initial directional guidance based on environmental snapshots, our method biases sampling toward regions more likely to contain feasible paths, significantly improving sampling efficiency and path quality. Extensive quantitative and qualitative experiments with various state-of-the-art VLMs demonstrate the effectiveness of this proposed approach.