# EchoTrack: Auditory Referring Multi-Object Tracking for Autonomous Driving arXiv: https://arxiv.org/abs/2402.18302 ยท HTML: https://mlsanity.com/paper/2402.18302/ ## Key facts - Cited by 19 papers in the map (data through 14 Oct 2026). - Placed in VLMs for autonomous driving, one of 9 directions in End-to-end autonomous driving. - Builds on 56 of its 90 references that are in the map. - First posted on arXiv 28 Feb 2024. ## Placed in - End-to-end autonomous driving / [VLMs for autonomous driving](https://mlsanity.com/driving/vlms-for-autonomous-driving/) ## Builds on (most cited first, 30 of 56) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020) - [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021) - [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018) - [YOLOX: Exceeding YOLO Series in 2021](https://mlsanity.com/paper/2107.08430/) (2021) - [Simple Online and Realtime Tracking with a Deep Association Metric](https://mlsanity.com/paper/1703.07402/) (2017) - [ByteTrack: Multi-Object Tracking by Associating Every Detection Box](https://mlsanity.com/paper/2110.06864/) (2021) - [HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking](https://mlsanity.com/paper/2009.07736/) (2020) - [Segment Everything Everywhere All at Once](https://mlsanity.com/paper/2304.06718/) (2023) - [MOT16: A Benchmark for Multi-Object Tracking](https://mlsanity.com/paper/1603.00831/) (2016) - [TrackFormer: Multi-Object Tracking with Transformers](https://mlsanity.com/paper/2101.02702/) (2021) - [MOTR: End-to-End Multiple-Object Tracking with Transformer](https://mlsanity.com/paper/2105.03247/) (2021) - [TransTrack: Multiple Object Tracking with Transformer](https://mlsanity.com/paper/2012.15460/) (2020) - [MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressions](https://mlsanity.com/paper/2308.08544/) (2023) - [Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentation](https://mlsanity.com/paper/2003.08813/) (2020) - [Universal Instance Perception as Object Discovery and Retrieval](https://mlsanity.com/paper/2303.06674/) (2023) - [Learning to Localize Sound Source in Visual Scenes](https://mlsanity.com/paper/1803.03849/) (2018) - [Localizing Visual Sounds the Hard Way](https://mlsanity.com/paper/2104.02691/) (2021) - [Audio-Visual Segmentation](https://mlsanity.com/paper/2207.05042/) (2022) - [MOTRv2: Bootstrapping End-to-End Multi-Object Tracking by Pretrained Object Detectors](https://mlsanity.com/paper/2211.09791/) (2022) - [PolyFormer: Referring Image Segmentation as Sequential Polygon Generation](https://mlsanity.com/paper/2302.07387/) (2023) - [Multiple Sound Sources Localization from Coarse to Fine](https://mlsanity.com/paper/2007.06355/) (2020) - [MeMOT: Multi-Object Tracking with Memory](https://mlsanity.com/paper/2203.16761/) (2022) - [GNN3DMOT: Graph Neural Network for 3D Multi-Object Tracking with Multi-Feature Learning](https://mlsanity.com/paper/2006.07327/) (2020) - [Language Prompt for Autonomous Driving](https://mlsanity.com/paper/2309.04379/) (2023) - [Global Tracking Transformers](https://mlsanity.com/paper/2203.13250/) (2022) - [Actor and Action Video Segmentation from a Sentence](https://mlsanity.com/paper/1803.07485/) (2018) - [Rethinking the competition between detection and ReID in Multi-Object Tracking](https://mlsanity.com/paper/2010.12138/) (2020) - [Referring Multi-Object Tracking](https://mlsanity.com/paper/2303.03366/) (2023) - [MeMOTR: Long-Term Memory-Augmented Transformer for Multi-Object Tracking](https://mlsanity.com/paper/2307.15700/) (2023) ## Built on by (most cited first, 19 of 19) - [MeViS: A Multi-Modal Dataset for Referring Motion Expression Video Segmentation](https://mlsanity.com/paper/2512.10945/) (2025) - [Bootstrapping Referring Multi-Object Tracking](https://mlsanity.com/paper/2406.05039/) (2024) - [Multimodal Referring Segmentation: A Survey](https://mlsanity.com/paper/2508.00265/) (2025) - [Referring Atomic Video Action Recognition](https://mlsanity.com/paper/2407.01872/) (2024) - [ReferGPT: Towards Zero-Shot Referring Multi-Object Tracking](https://mlsanity.com/paper/2504.09195/) (2025) - [Cognitive Disentanglement for Referring Multi-Object Tracking](https://mlsanity.com/paper/2503.11496/) (2025) - [Motion-Aware Vision-Reference Alignment for Referring Multi-Object Tracking](https://mlsanity.com/paper/2511.17681/) (2025) - [STORM: End-to-End Referring Multi-Object Tracking in Videos](https://mlsanity.com/paper/2604.10527/) (2026) - [Global-Local Distillation Network-Based Audio-Visual Speaker Tracking with Incomplete Modalities](https://mlsanity.com/paper/2408.14585/) (2024) - [RefAtomNet++: Advancing Referring Atomic Video Action Recognition using Semantic Retrieval based Multi-Trajectory Mamba](https://mlsanity.com/paper/2510.16444/) (2025) - [Cross-modal Context-aware Learning for Visual Prompt Guided Multimodal Image Understanding in Remote Sensing](https://mlsanity.com/paper/2512.11680/) (2025) - [Scene Graph-guided SegCaptioning Transformer with Fine-grained Alignment for Controllable Video Segmentation and Captioning](https://mlsanity.com/paper/2603.20887/) (2026) - [COAL: Counterfactual and Observation-Enhanced Alignment Learning for Discriminative Referring Multi-Object Tracking](https://mlsanity.com/paper/2605.14795/) (2026) - [Learning to Label: A Reinforced Self-Evolving Framework for Semi-supervised Referring Expression Segmentation](https://mlsanity.com/paper/2605.28239/) (2026) - [FeVOS: Foresight Expression Video Object Segmentation](https://mlsanity.com/paper/2606.25585/) (2026) - [CD-RMOT-Bench: Benchmarking the Cross-Domain Referring Multi-Object Tracking](https://mlsanity.com/paper/2607.25239/) (2026) - [A Collaborative Multi-Modality Interaction for VLA-based End-to-End Autonomous Driving](https://mlsanity.com/paper/2608.20890/) (2026) - [YesTrack: Referring Multi-Object Tracking via MLLM-based Yes/No Verification](https://mlsanity.com/paper/2609.02318/) (2026) - [SlotAVS: Object-Centric Audio-Visual Segmentation via Bi-Modal Slot Attention](https://mlsanity.com/paper/2610.15764/) (2026) ## Abstract (arXiv) This paper introduces the task of Auditory Referring Multi-Object Tracking (AR-MOT), which dynamically tracks specific objects in a video sequence based on audio expressions and appears as a challenging problem in autonomous driving. Due to the lack of semantic modeling capacity in audio and video, existing works have mainly focused on text-based multi-object tracking, which often comes at the cost of tracking quality, interaction efficiency, and even the safety of assistance systems, limiting the application of such methods in autonomous driving. In this paper, we delve into the problem of AR-MOT from the perspective of audio-video fusion and audio-video tracking. We put forward EchoTrack, an end-to-end AR-MOT framework with dual-stream vision transformers. The dual streams are intertwined with our Bidirectional Frequency-domain Cross-attention Fusion Module (Bi-FCFM), which bidirectionally fuses audio and video features from both frequency- and spatiotemporal domains. Moreover, we propose the Audio-visual Contrastive Tracking Learning (ACTL) regime to extract homogeneous semantic features between expressions and visual objects by learning homogeneous features between different audio and video objects effectively. Aside from the architectural design, we establish the first set of large-scale AR-MOT benchmarks, including Echo-KITTI, Echo-KITTI+, and Echo-BDD. Extensive experiments on the established benchmarks demonstrate the effectiveness of the proposed EchoTrack and its components. The source code and datasets are available at https://github.com/lab206/EchoTrack.