# Can Synthetic UAV Benchmarks Be Trusted? Real-Synthetic Validation for UAV Model Selection

arXiv: https://arxiv.org/abs/2610.14647 · HTML: https://mlsanity.com/paper/2610.14647/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 29 references that are in the map.
- First posted on arXiv 10 Oct 2026.

## Builds on (most cited first, 25 of 25)

- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [Simple Open-Vocabulary Object Detection with Vision Transformers](https://mlsanity.com/paper/2205.06230/) (2022)
- [Detection and Tracking Meet Drones Challenge](https://mlsanity.com/paper/2001.06303/) (2020)
- [RF-DETR: Neural Architecture Search for Real-Time Detection Transformers](https://mlsanity.com/paper/2511.09554/) (2025)
- [Detect Anything via Next Point Prediction](https://mlsanity.com/paper/2510.12798/) (2025)
- [SeaDronesSee: A Maritime Benchmark for Detecting Humans in Open Water](https://mlsanity.com/paper/2105.01922/) (2021)
- [AutoEval Done Right: Using Synthetic Data for Model Evaluation](https://mlsanity.com/paper/2403.07008/) (2024)
- [AU-AIR: A Multi-modal Unmanned Aerial Vehicle Dataset for Low Altitude Traffic Surveillance](https://mlsanity.com/paper/2001.11737/) (2020)
- [AeroGen: Enhancing Remote Sensing Object Detection with Diffusion-Driven Data Generation](https://mlsanity.com/paper/2411.15497/) (2024)
- [HIT-UAV: A high-altitude infrared thermal dataset for Unmanned Aerial Vehicle-based object detection](https://mlsanity.com/paper/2204.03245/) (2022)
- [Can You Rely on Your Model Evaluation? Improving Model Evaluation with Synthetic Test Data](https://mlsanity.com/paper/2310.16524/) (2023)
- [Task Me Anything](https://mlsanity.com/paper/2406.11775/) (2024)
- [SynDrone -- Multi-modal UAV Dataset for Urban Scenarios](https://mlsanity.com/paper/2308.10491/) (2023)
- [LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decoding](https://mlsanity.com/paper/2605.27365/) (2026)
- [UAVScenes: A Multi-Modal Dataset for UAVs](https://mlsanity.com/paper/2507.22412/) (2025)
- [SkyScenes: A Synthetic Dataset for Aerial Scene Understanding](https://mlsanity.com/paper/2312.06719/) (2023)
- [Archangel: A Hybrid UAV-based Human Detection Benchmark with Position and Pose Metadata](https://mlsanity.com/paper/2209.00128/) (2022)
- [Synthetic Data for Model Selection](https://mlsanity.com/paper/2105.00717/) (2021)
- [Fusion Meets Diverse Conditions: A High-diversity Benchmark and Baseline for UAV-based Multimodal Object Detection with Condition Cues](https://mlsanity.com/paper/2510.13620/) (2025)
- [Validation of object detection in UAV-based images using synthetic data](https://mlsanity.com/paper/2201.06629/) (2022)
- [CrowdSim2: an Open Synthetic Benchmark for Object Detectors](https://mlsanity.com/paper/2304.05090/) (2023)
- [Visual Prototype Conditioned Focal Region Generation for UAV-Based Object Detection](https://mlsanity.com/paper/2604.02966/) (2026)

## Abstract (arXiv)

Synthetic UAV imagery can expand pre-deployment test evidence when task-specific Real data are scarce, but visual plausibility or semantic label alignment alone does not establish validity for model selection. We study when Synthetic evaluation preserves the model-selection conclusions obtained from Real UAV data through two complementary cases under fixed evaluation protocols. In a requirement-specific road-debris benchmark, four language-conditioned perception systems obtain lower absolute F1 on Synthetic data, yet the point-estimate ordering and top-ranked model are preserved; bootstrap analysis yields no supported pairwise reversal, and size-matched resampling preserves the top-model decision. In contrast, a closed-set stress test on a separately materialized business corpus, using four VisDrone-compatible detectors over eight empirically supported classes, changes the point-estimate top-ranked detector and reverses three of six pairwise point-estimate preference directions. A complementary audit of five UAV business scenarios further reveals substantial Real-Synthetic mismatch in class composition, object density, and target scale, including systematic under-representation of small targets. These results show that Synthetic UAV benchmark trust is conditional: semantic coverage is insufficient, and validity must be established empirically for the task, operating regime, and model panel of interest.