# AV-DTEC: Self-Supervised Audio-Visual Fusion for Drone Trajectory Estimation and Classification

arXiv: https://arxiv.org/abs/2412.16928 · HTML: https://mlsanity.com/paper/2412.16928/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 31 references that are in the map.
- First posted on arXiv 22 Dec 2024.

## Builds on (most cited first, 8 of 8)

- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model](https://mlsanity.com/paper/2401.09417/) (2024)
- [Anti-UAV: A Large Multi-Modal Benchmark for UAV Tracking](https://mlsanity.com/paper/2101.08466/) (2021)
- [The 2nd Anti-UAV Workshop & Challenge: Methods and Results](https://mlsanity.com/paper/2108.09909/) (2021)
- [The 3rd Anti-UAV Workshop & Challenge: Methods and Results](https://mlsanity.com/paper/2305.07290/) (2023)
- [Multi-Modal UAV Detection, Classification and Tracking Algorithm -- Technical Report for CVPR 2024 UG2 Challenge](https://mlsanity.com/paper/2405.16464/) (2024)
- [Clustering-based Learning for UAV Tracking and Pose Estimation](https://mlsanity.com/paper/2405.16867/) (2024)

## Built on by (most cited first, 1 of 1)

- [Securing the Skies: A Comprehensive Survey on Anti-UAV Methods, Benchmarking, and Future Directions](https://mlsanity.com/paper/2504.11967/) (2025)

## Abstract (arXiv)

The increasing use of compact UAVs has created significant threats to public safety, while traditional drone detection systems are often bulky and costly. To address these challenges, we propose AV-DTEC, a lightweight self-supervised audio-visual fusion-based anti-UAV system. AV-DTEC is trained using self-supervised learning with labels generated by LiDAR, and it simultaneously learns audio and visual features through a parallel selective state-space model. With the learned features, a specially designed plug-and-play primary-auxiliary feature enhancement module integrates visual features into audio features for better robustness in cross-lighting conditions. To reduce reliance on auxiliary features and align modalities, we propose a teacher-student model that adaptively adjusts the weighting of visual features. AV-DTEC demonstrates exceptional accuracy and effectiveness in real-world multi-modality data. The code and trained models are publicly accessible on GitHub urlhttps://github.com/AmazingDay1/AV-DETC.