# A Safer Vision-based Autonomous Planning System for Quadrotor UAVs with Dynamic Obstacle Trajectory Prediction and Its Application with LLMs

arXiv: https://arxiv.org/abs/2311.12893 · HTML: https://mlsanity.com/paper/2311.12893/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 53 references that are in the map.
- First posted on arXiv 21 Nov 2023.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016)
- [LLM+P: Empowering Large Language Models with Optimal Planning Proficiency](https://mlsanity.com/paper/2304.11477/) (2023)
- [ChatGPT for Robotics: Design Principles and Model Abilities](https://mlsanity.com/paper/2306.17582/) (2023)
- [Stepwise Goal-Driven Networks for Trajectory Prediction](https://mlsanity.com/paper/2103.14107/) (2021)
- [EGO-Planner: An ESDF-free Gradient-based Local Planner for Quadrotors](https://mlsanity.com/paper/2008.08835/) (2020)
- [ChatGPT Empowered Long-Step Robot Control in Various Environments: A Case Application](https://mlsanity.com/paper/2304.03893/) (2023)
- [A General Optimization-based Framework for Local Odometry Estimation with Multiple Sensors](https://mlsanity.com/paper/1901.03638/) (2019)
- [Robust and Efficient Quadrotor Trajectory Generation for Fast Autonomous Flight](https://mlsanity.com/paper/1907.01531/) (2019)
- [A Data-driven Model for Interaction-aware Pedestrian Motion Prediction in Object Cluttered Environments](https://mlsanity.com/paper/1709.08528/) (2017)
- [Towards Monocular Vision based Obstacle Avoidance through Deep Reinforcement Learning](https://mlsanity.com/paper/1706.09829/) (2017)
- [MADER: Trajectory Planner in Multi-Agent and Dynamic Environments](https://mlsanity.com/paper/2010.11061/) (2020)
- [FASTER: Fast and Safe Trajectory Planner for Flights in Unknown Environments](https://mlsanity.com/paper/1903.03558/) (2019)
- [Robust Vision-based Obstacle Avoidance for Micro Aerial Vehicles in Dynamic Environments](https://mlsanity.com/paper/2002.04920/) (2020)
- [Leveraging Stereo-Camera Data for Real-Time Dynamic Obstacle Detection and Tracking](https://mlsanity.com/paper/2007.10743/) (2020)
- [Autonomous Flights in Dynamic Environments with Onboard Vision](https://mlsanity.com/paper/2103.05870/) (2021)
- [Real-Time Trajectory Replanning for MAVs using Uniform B-splines and a 3D Circular Buffer](https://mlsanity.com/paper/1703.01416/) (2017)
- [Risk-aware Trajectory Sampling for Quadrotor Obstacle Avoidance in Dynamic Environments](https://mlsanity.com/paper/2201.06645/) (2022)
- [ChatSim: Underwater Simulation with Natural Language Prompting](https://mlsanity.com/paper/2308.04029/) (2023)
- [A real-time dynamic obstacle tracking and mapping system for UAV navigation and collision avoidance with an RGB-D camera](https://mlsanity.com/paper/2209.08258/) (2022)
- [Real-time Trajectory Generation for Quadrotors using B-spline based Non-uniform Kinodynamic Search](https://mlsanity.com/paper/1904.12348/) (2019)

## Built on by (most cited first, 13 of 13)

- [Towards Realistic UAV Vision-Language Navigation: Platform, Benchmark, and Methodology](https://mlsanity.com/paper/2410.07087/) (2024)
- [Large Language Model Enhanced Multi-Agent Systems for 6G Communications](https://mlsanity.com/paper/2312.07850/) (2023)
- [UAVs Meet LLMs: Overviews and Perspectives Toward Agentic Low-Altitude Mobility](https://mlsanity.com/paper/2501.02341/) (2025)
- [Large Language Models for UAVs: Current State and Pathways to the Future](https://mlsanity.com/paper/2405.01745/) (2024)
- [UAV-VLA: Vision-Language-Action System for Large Scale Aerial Mission Generation](https://mlsanity.com/paper/2501.05014/) (2025)
- [RaceVLA: VLA-based Racing Drone Navigation with Human-like Behaviour](https://mlsanity.com/paper/2503.02572/) (2025)
- [CognitiveDrone: A VLA Model and Evaluation Benchmark for Real-Time Cognitive Task Solving and Reasoning in UAVs](https://mlsanity.com/paper/2503.01378/) (2025)
- [Next-Generation LLM for UAV: From Natural Language to Autonomous Flight](https://mlsanity.com/paper/2510.21739/) (2025)
- [From Large AI Models to Agentic AI: A Tutorial on Future Intelligent Communications](https://mlsanity.com/paper/2505.22311/) (2025)
- [An LLM-based Framework for Human-Swarm Teaming Cognition in Disaster Search and Rescue](https://mlsanity.com/paper/2511.04042/) (2025)
- [Safe and Economical UAV Trajectory Planning in Low-Altitude Airspace: A Hybrid DRL-LLM Approach with Compliance Awareness](https://mlsanity.com/paper/2506.08532/) (2025)
- [Self-Supervised Learning to Fly using Efficient Semantic Segmentation and Metric Depth Estimation for Low-Cost Autonomous UAVs](https://mlsanity.com/paper/2510.16624/) (2025)
- [No Training, Better Flights: Test-Time Scaled VLMs for UAV Navigation](https://mlsanity.com/paper/2607.19288/) (2026)

## Abstract (arXiv)

For intelligent quadcopter UAVs, a robust and reliable autonomous planning system is crucial. Most current trajectory planning methods for UAVs are suitable for static environments but struggle to handle dynamic obstacles, which can pose challenges and even dangers to flight. To address this issue, this paper proposes a vision-based planning system that combines tracking and trajectory prediction of dynamic obstacles to achieve efficient and reliable autonomous flight. We use a lightweight object detection algorithm to identify dynamic obstacles and then use Kalman Filtering to track and estimate their motion states. During the planning phase, we not only consider static obstacles but also account for the potential movements of dynamic obstacles. For trajectory generation, we use a B-spline-based trajectory search algorithm, which is further optimized with various constraints to enhance safety and alignment with the UAV's motion characteristics. We conduct experiments in both simulation and real-world environments, and the results indicate that our approach can successfully detect and avoid obstacles in dynamic environments in real-time, offering greater reliability compared to existing approaches. Furthermore, with the advancements in Natural Language Processing (NLP) technology demonstrating exceptional zero-shot generalization capabilities, more user-friendly human-machine interactions have become feasible, and this study also explores the integration of autonomous planning systems with Large Language Models (LLMs).