# Multi-Modal UAV Detection, Classification and Tracking Algorithm -- Technical Report for CVPR 2024 UG2 Challenge

arXiv: https://arxiv.org/abs/2405.16464 · HTML: https://mlsanity.com/paper/2405.16464/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 61 references that are in the map.
- First posted on arXiv 26 May 2024.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors](https://mlsanity.com/paper/2207.02696/) (2022)
- [YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information](https://mlsanity.com/paper/2402.13616/) (2024)
- [SGS-SLAM: Semantic Gaussian Splatting For Neural Dense SLAM](https://mlsanity.com/paper/2402.03246/) (2024)
- [Point Mamba: A Novel Point Cloud Backbone Based on State Space Model with Octree-Based Ordering Strategy](https://mlsanity.com/paper/2403.06467/) (2024)
- [SNI-SLAM: Semantic Neural Implicit SLAM](https://mlsanity.com/paper/2311.11016/) (2023)
- [MotionTrack: Learning Robust Short-term and Long-term Motions for Multi-Object Tracking](https://mlsanity.com/paper/2303.10404/) (2023)
- [PLGSLAM: Progressive Neural Scene Represenation with Local to Global Bundle Adjustment](https://mlsanity.com/paper/2312.09866/) (2023)
- [VMT-Adapter: Parameter-Efficient Transfer Learning for Multi-Task Dense Scene Understanding](https://mlsanity.com/paper/2312.08733/) (2023)
- [Deception Detection from Linguistic and Physiological Data Streams Using Bimodal Convolutional Neural Networks](https://mlsanity.com/paper/2311.10944/) (2023)
- [Contextual Hourglass Network for Semantic Segmentation of High Resolution Aerial Imagery](https://mlsanity.com/paper/1810.12813/) (2018)
- [MmAP : Multi-modal Alignment Prompt for Cross-domain Multi-task Learning](https://mlsanity.com/paper/2312.08636/) (2023)
- [RegFormer: An Efficient Projection-Aware Transformer Network for Large-Scale Point Cloud Registration](https://mlsanity.com/paper/2303.12384/) (2023)
- [LIF-Seg: LiDAR and Camera Image Fusion for 3D LiDAR Semantic Segmentation](https://mlsanity.com/paper/2108.07511/) (2021)
- [NeSLAM: Neural Implicit Mapping and Self-Supervised Feature Tracking With Depth Completion and Denoising](https://mlsanity.com/paper/2403.20034/) (2024)
- [Regional Style and Color Transfer](https://mlsanity.com/paper/2404.13880/) (2024)
- [Particle Filter SLAM for Vehicle Localization](https://mlsanity.com/paper/2402.07429/) (2024)
- [Confidence Trigger Detection: Accelerating Real-time Tracking-by-detection Systems](https://mlsanity.com/paper/1902.00615/) (2019)
- [MotionTrack: End-to-End Transformer-based Multi-Object Tracing with LiDAR-Camera Fusion](https://mlsanity.com/paper/2306.17000/) (2023)
- [Enhanced Detection Classification via Clustering SVM for Various Robot Collaboration Task](https://mlsanity.com/paper/2405.03026/) (2024)
- [MMAUD: A Comprehensive Multi-Modal Anti-UAV Dataset for Modern Miniature Drone Threats](https://mlsanity.com/paper/2402.03706/) (2024)
- [Zero-shot Synthesis with Group-Supervised Learning](https://mlsanity.com/paper/2009.06586/) (2020)
- [Cross-Task Multi-Branch Vision Transformer for Facial Expression and Mask Wearing Classification](https://mlsanity.com/paper/2404.14606/) (2024)
- [Rumor Detection with a novel graph neural network approach](https://mlsanity.com/paper/2403.16206/) (2024)
- [Incremental Joint Learning of Depth, Pose and Implicit Scene Representation on Monocular Camera in Large-scale Scenes](https://mlsanity.com/paper/2404.06050/) (2024)
- [Adaptive speed planning for Unmanned Vehicle Based on Deep Reinforcement Learning](https://mlsanity.com/paper/2404.17379/) (2024)
- [SupFusion: Supervised LiDAR-Camera Fusion for 3D Object Detection](https://mlsanity.com/paper/2309.07084/) (2023)
- [Vehicle Pose and Shape Estimation through Multiple Monocular Vision](https://mlsanity.com/paper/1802.03515/) (2018)
- [DeepGI: An Automated Approach for Gastrointestinal Tract Segmentation in MRI Scans](https://mlsanity.com/paper/2401.15354/) (2024)

## Built on by (most cited first, 9 of 9)

- [Time Series Modeling for Heart Rate Prediction: From ARIMA to Transformers](https://mlsanity.com/paper/2406.12199/) (2024)
- [Harnessing Earnings Reports for Stock Predictions: A QLoRA-Enhanced LLM Approach](https://mlsanity.com/paper/2408.06634/) (2024)
- [AR Overlay: Training Image Pose Estimation on Curved Surface in a Synthetic Way](https://mlsanity.com/paper/2409.14577/) (2024)
- [SAM-Guided Masked Token Prediction for 3D Scene Understanding](https://mlsanity.com/paper/2410.12158/) (2024)
- [TAME: Temporal Audio-based Mamba for Enhanced Drone Trajectory Estimation and Classification](https://mlsanity.com/paper/2412.13037/) (2024)
- [UNO: Unified Self-Supervised Monocular Odometry for Platform-Agnostic Deployment](https://mlsanity.com/paper/2506.07013/) (2025)
- [AV-DTEC: Self-Supervised Audio-Visual Fusion for Drone Trajectory Estimation and Classification](https://mlsanity.com/paper/2412.16928/) (2024)
- [Enhancing Low-Altitude Airspace Security: MLLM-Enabled UAV Intent Recognition](https://mlsanity.com/paper/2509.06312/) (2025)
- [Efficient UAV trajectory prediction: A multi-modal deep diffusion framework](https://mlsanity.com/paper/2602.00107/) (2026)

## Abstract (arXiv)

This technical report presents the 1st winning model for UG2+, a task in CVPR 2024 UAV Tracking and Pose-Estimation Challenge. This challenge faces difficulties in drone detection, UAV-type classification and 2D/3D trajectory estimation in extreme weather conditions with multi-modal sensor information, including stereo vision, various Lidars, Radars, and audio arrays. Leveraging this information, we propose a multi-modal UAV detection, classification, and 3D tracking method for accurate UAV classification and tracking. A novel classification pipeline which incorporates sequence fusion, region of interest (ROI) cropping, and keyframe selection is proposed. Our system integrates cutting-edge classification techniques and sophisticated post-processing steps to boost accuracy and robustness. The designed pose estimation pipeline incorporates three modules: dynamic points analysis, a multi-object tracker, and trajectory completion techniques. Extensive experiments have validated the effectiveness and precision of our approach. In addition, we also propose a novel dataset pre-processing method and conduct a comprehensive ablation study for our design. We finally achieved the best performance in the classification and tracking of the MMUAD dataset. The code and configuration of our method are available at https://github.com/dtc111111/Multi-Modal-UAV.