# AerialMind: Towards Referring Multi-Object Tracking in UAV Scenarios

arXiv: https://arxiv.org/abs/2511.21053 · HTML: https://mlsanity.com/paper/2511.21053/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 43 references that are in the map.
- First posted on arXiv 26 Nov 2025.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Modeling Context in Referring Expressions](https://mlsanity.com/paper/1608.00272/) (2016)
- [HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking](https://mlsanity.com/paper/2009.07736/) (2020)
- [TransTrack: Multiple Object Tracking with Transformer](https://mlsanity.com/paper/2012.15460/) (2020)
- [MOSE: A New Dataset for Video Object Segmentation in Complex Scenes](https://mlsanity.com/paper/2302.01872/) (2023)
- [The Unmanned Aerial Vehicle Benchmark: Object Detection and Tracking](https://mlsanity.com/paper/1804.00518/) (2018)
- [Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentation](https://mlsanity.com/paper/2003.08813/) (2020)
- [Video Object Segmentation with Language Referring Expressions](https://mlsanity.com/paper/1803.08006/) (2018)
- [MOTRv2: Bootstrapping End-to-End Multi-Object Tracking by Pretrained Object Detectors](https://mlsanity.com/paper/2211.09791/) (2022)
- [A Survey on Deep Learning Technique for Video Segmentation](https://mlsanity.com/paper/2107.01153/) (2021)
- [A Real-Time Cross-modality Correlation Filtering Method for Referring Expression Comprehension](https://mlsanity.com/paper/1909.07072/) (2019)
- [VLT: Vision-Language Transformer and Query Generation for Referring Segmentation](https://mlsanity.com/paper/2210.15871/) (2022)
- [MOSEv2: A More Challenging Dataset for Video Object Segmentation in Complex Scenes](https://mlsanity.com/paper/2508.05630/) (2025)
- [Comprehension-guided referring expressions](https://mlsanity.com/paper/1701.03439/) (2017)
- [Local-Global Context Aware Transformer for Language-Guided Video Segmentation](https://mlsanity.com/paper/2203.09773/) (2022)
- [Collaborative Spatial-Temporal Modeling for Language-Queried Video Actor Segmentation](https://mlsanity.com/paper/2105.06818/) (2021)
- [iKUN: Speak to Trackers without Retraining](https://mlsanity.com/paper/2312.16245/) (2023)
- [UAV-Flow Colosseo: A Real-World Benchmark for Flying-on-a-Word UAV Imitation Learning](https://mlsanity.com/paper/2505.15725/) (2025)
- [SATO: Stable Text-to-Motion Framework](https://mlsanity.com/paper/2405.01461/) (2024)
- [RefDrone: A Challenging Benchmark for Referring Expression Comprehension in Drone Scenes](https://mlsanity.com/paper/2502.00392/) (2025)
- [Delving into the Trajectory Long-tail Distribution for Muti-object Tracking](https://mlsanity.com/paper/2403.04700/) (2024)
- [Talk2Radar: Bridging Natural Language with 4D mmWave Radar for 3D Referring Expression Comprehension](https://mlsanity.com/paper/2405.12821/) (2024)
- [WaterVG: Waterway Visual Grounding based on Text-Guided Vision and mmWave Radar](https://mlsanity.com/paper/2403.12686/) (2024)
- [AerialVG: A Challenging Benchmark for Aerial Visual Grounding by Exploring Positional Relations](https://mlsanity.com/paper/2504.07836/) (2025)
- [ANT: Adaptive Neural Temporal-Aware Text-to-Motion Model](https://mlsanity.com/paper/2506.02452/) (2025)
- [MLS-Track: Multilevel Semantic Interaction in RMOT](https://mlsanity.com/paper/2404.12031/) (2024)
- [NanoMVG: USV-Centric Low-Power Multi-Task Visual Grounding based on Prompt-Guided Camera and 4D mmWave Radar](https://mlsanity.com/paper/2408.17207/) (2024)
- [Learning New Concepts, Remembering the Old: Continual Learning for Multimodal Concept Bottleneck Models](https://mlsanity.com/paper/2411.17471/) (2024)
- [WinDB: HMD-free and Distortion-free Panoptic Video Fixation Learning](https://mlsanity.com/paper/2305.13901/) (2023)

## Built on by (most cited first, 4 of 4)

- [CD-RMOT-Bench: Benchmarking the Cross-Domain Referring Multi-Object Tracking](https://mlsanity.com/paper/2607.25239/) (2026)
- [When Efficiency Becomes Fragility: Exploiting Dynamic Routing Vulnerabilities in Adaptive UAV Tracking](https://mlsanity.com/paper/2608.03902/) (2026)
- [GrabVG: Graph-Attentive Binding for Visual Grounding in UAV Imagery](https://mlsanity.com/paper/2608.18996/) (2026)
- [The Gap Is in the Geometry: A Convergent Meta-Analysis of Multi-Facet Aerial and Remote Sensing Video Understanding](https://mlsanity.com/paper/2610.19102/) (2026)

## Abstract (arXiv)

Referring Multi-Object Tracking (RMOT) aims to achieve precise object detection and tracking through natural language instructions, representing a fundamental capability for intelligent robotic systems. However, current RMOT research remains mostly confined to ground-level scenarios, which constrains their ability to capture broad-scale scene contexts and perform comprehensive tracking and path planning. In contrast, Unmanned Aerial Vehicles (UAVs) leverage their expansive aerial perspectives and superior maneuverability to enable wide-area surveillance. Moreover, UAVs have emerged as critical platforms for Embodied Intelligence, which has given rise to an unprecedented demand for intelligent aerial systems capable of natural language interaction. To this end, we introduce AerialMind, the first large-scale RMOT benchmark in UAV scenarios, which aims to bridge this research gap. To facilitate its construction, we develop an innovative semi-automated collaborative agent-based labeling assistant (COALA) framework that significantly reduces labor costs while maintaining annotation quality. Furthermore, we propose HawkEyeTrack (HETrack), a novel method that collaboratively enhances vision-language representation learning and improves the perception of UAV scenarios. Comprehensive experiments validated the challenging nature of our dataset and the effectiveness of our method.