# LLM-Agents Driven Automated Simulation Testing and Analysis of small Uncrewed Aerial Systems arXiv: https://arxiv.org/abs/2501.11864 ยท HTML: https://mlsanity.com/paper/2501.11864/ ## Key facts - Cited by 4 papers in the map (data through 14 Oct 2026). - Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. - Builds on 24 of its 71 references that are in the map. - First posted on arXiv 21 Jan 2025. ## Placed in - LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/) ## Builds on (most cited first, 24 of 24) - [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020) - [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017) - [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022) - [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024) - [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022) - [A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT](https://mlsanity.com/paper/2302.11382/) (2023) - [AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles](https://mlsanity.com/paper/1705.05065/) (2017) - [NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models](https://mlsanity.com/paper/2405.17428/) (2024) - [Ragas: Automated Evaluation of Retrieval Augmented Generation](https://mlsanity.com/paper/2309.15217/) (2023) - [ChatScene: Knowledge-Enabled Safety-Critical Scenario Generation for Autonomous Vehicles](https://mlsanity.com/paper/2405.14062/) (2024) - [Review of Large Vision Models and Visual Prompt Engineering](https://mlsanity.com/paper/2307.00855/) (2023) - [Unleashing the potential of prompt engineering for large language models](https://mlsanity.com/paper/2310.14735/) (2023) - [Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation with Large Language Models](https://mlsanity.com/paper/2208.07852/) (2022) - [RobOT: Robustness-Oriented Testing for Deep Learning Systems](https://mlsanity.com/paper/2102.05913/) (2021) - [Generating Test Scenarios from NL Requirements using Retrieval-Augmented LLMs: An Industrial Study](https://mlsanity.com/paper/2404.12772/) (2024) - [The Next Generation of Human-Drone Partnerships: Co-Designing an Emergency Response System](https://mlsanity.com/paper/2001.03849/) (2020) - [Aegis:An Advanced LLM-Based Multi-Agent for Intelligent Functional Safety Engineering](https://mlsanity.com/paper/2410.12475/) (2024) - [RESAM: Requirements Elicitation and Specification for Deep-Learning Anomaly Models with Applications to UAV Flight Controllers](https://mlsanity.com/paper/2207.08857/) (2022) - [OpenUAV Cloud Testbed: a Collaborative Design Studio for Field Robotics](https://mlsanity.com/paper/1910.00739/) (2019) - [DroneReqValidator: Facilitating High Fidelity Simulation Testing for Uncrewed Aerial Systems Developers](https://mlsanity.com/paper/2308.00174/) (2023) - [Coupled Requirements-driven Testing of CPS: From Simulation To Reality](https://mlsanity.com/paper/2403.16287/) (2024) - [A Requirements-Driven Platform for Validating Field Operations of Small Uncrewed Aerial Vehicles](https://mlsanity.com/paper/2307.00194/) (2023) - [UAV-Assisted Maritime Search and Rescue: A Holistic Approach](https://mlsanity.com/paper/2403.14281/) (2024) - [DroneWiS: Automated Simulation Testing of small Unmanned Aerial Systems in Realistic Windy Conditions](https://mlsanity.com/paper/2408.16559/) (2024) ## Built on by (most cited first, 4 of 4) - [Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system security](https://mlsanity.com/paper/2605.23989/) (2026) - [AutoMT: A Multi-Agent LLM Framework for Automated Metamorphic Testing of Autonomous Driving Systems](https://mlsanity.com/paper/2510.19438/) (2025) - [An LLM-Agnostic, MAVLink-Based Drone Command and Control Interface and Agentic Harness Using the Model Context Protocol](https://mlsanity.com/paper/2601.15486/) (2026) - [Reproducible, Explainable, and Effective Evaluations of Agentic AI for Software Engineering](https://mlsanity.com/paper/2604.01437/) (2026) ## Abstract (arXiv) Thorough simulation testing is crucial for validating the correct behavior of small Uncrewed Aerial Systems (sUAS) across multiple scenarios, including adverse weather conditions (such as wind, and fog), diverse settings (hilly terrain, or urban areas), and varying mission profiles (surveillance, tracking). While various sUAS simulation tools exist to support developers, the entire process of creating, executing, and analyzing simulation tests remains a largely manual and cumbersome task. Developers must identify test scenarios, set up the simulation environment, integrate the System under Test (SuT) with simulation tools, formulate mission plans, and collect and analyze results. These labor-intensive tasks limit the ability of developers to conduct exhaustive testing across a wide range of scenarios. To alleviate this problem, in this paper, we propose AutoSimTest, a Large Language Model (LLM)-driven framework, where multiple LLM agents collaborate to support the sUAS simulation testing process. This includes: (1) creating test scenarios that subject the SuT to unique environmental contexts; (2) preparing the simulation environment as per the test scenario; (3) generating diverse sUAS missions for the SuT to execute; and (4) analyzing simulation results and providing an interactive analytics interface. Further, the design of the framework is flexible for creating and testing scenarios for a variety of sUAS use cases, simulation tools, and SuT input requirements. We evaluated our approach by (a) conducting simulation testing of PX4 and ArduPilot flight-controller-based SuTs, (b) analyzing the performance of each agent, and (c) gathering feedback from sUAS developers. Our findings indicate that AutoSimTest significantly improves the efficiency and scope of the sUAS testing process, allowing for more comprehensive and varied scenario evaluations while reducing the manual effort.