# Self-Supervised Moving Vehicle Detection from Audio-Visual Cues

arXiv: https://arxiv.org/abs/2201.12771 · HTML: https://mlsanity.com/paper/2201.12771/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 27 references that are in the map.
- First posted on arXiv 30 Jan 2022.

## Builds on (most cited first, 19 of 19)

- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [EfficientDet: Scalable and Efficient Object Detection](https://mlsanity.com/paper/1911.09070/) (2019)
- [Deep Visual Domain Adaptation: A Survey](https://mlsanity.com/paper/1802.03601/) (2018)
- [Audio-Visual Scene Analysis with Self-Supervised Multisensory Features](https://mlsanity.com/paper/1804.03641/) (2018)
- [Objects that Sound](https://mlsanity.com/paper/1712.06651/) (2017)
- [Learning to Localize Sound Source in Visual Scenes](https://mlsanity.com/paper/1803.03849/) (2018)
- [Localizing Visual Sounds the Hard Way](https://mlsanity.com/paper/2104.02691/) (2021)
- [Multiple Sound Sources Localization from Coarse to Fine](https://mlsanity.com/paper/2007.06355/) (2020)
- [Discriminative Sounding Objects Localization via Self-supervised Audiovisual Matching](https://mlsanity.com/paper/2010.05466/) (2020)
- [Deep Visual Domain Adaptation](https://mlsanity.com/paper/2012.14176/) (2020)
- [Sound Event Localization and Detection of Overlapping Sources Using Convolutional Recurrent Neural Networks](https://mlsanity.com/paper/1807.00129/) (2018)
- [Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input](https://mlsanity.com/paper/1804.01452/) (2018)
- [Self-supervised Moving Vehicle Tracking with Stereo Sound](https://mlsanity.com/paper/1910.11760/) (2019)
- [There is More than Meets the Eye: Self-Supervised Multi-Object Detection and Tracking with Sound by Distilling Multimodal Knowledge](https://mlsanity.com/paper/2103.01353/) (2021)
- [HeatNet: Bridging the Day-Night Domain Gap in Semantic Segmentation with Thermal Images](https://mlsanity.com/paper/2003.04645/) (2020)
- [Self-Supervised Visual Terrain Classification from Unsupervised Acoustic Feature Learning](https://mlsanity.com/paper/1912.03227/) (2019)
- [Self-supervised object detection from audio-visual correspondence](https://mlsanity.com/paper/2104.06401/) (2021)
- [Multimodal Interaction-aware Motion Prediction for Autonomous Street Crossing](https://mlsanity.com/paper/1808.06887/) (2018)
- [Why did the Robot Cross the Road? - Learning from Multi-Modal Sensor Data for Autonomous Road Crossing](https://mlsanity.com/paper/1709.06039/) (2017)

## Built on by (most cited first, 8 of 8)

- [Uncertainty-aware Panoptic Segmentation](https://mlsanity.com/paper/2206.14554/) (2022)
- [Joint Audio-Visual Idling Vehicle Detection with Streamlined Input Dependencies](https://mlsanity.com/paper/2410.21170/) (2024)
- [Can Textual Semantics Mitigate Sounding Object Segmentation Preference?](https://mlsanity.com/paper/2407.10947/) (2024)
- [Unveiling and Mitigating Bias in Audio Visual Segmentation](https://mlsanity.com/paper/2407.16638/) (2024)
- [Segment Beyond View: Handling Partially Missing Modality for Audio-Visual Semantic Segmentation](https://mlsanity.com/paper/2312.08673/) (2023)
- [TrackletMapper: Ground Surface Segmentation and Mapping from Traffic Participant Trajectories](https://mlsanity.com/paper/2209.05247/) (2022)
- [SpA2V: Harnessing Spatial Auditory Cues for Audio-driven Spatially-aware Video Generation](https://mlsanity.com/paper/2508.00782/) (2025)
- [Evidence of Absence: Cross-Modal Abductive Risk Perception to Sustain World Models When Vision Fails](https://mlsanity.com/paper/2608.14952/) (2026)

## Abstract (arXiv)

Robust detection of moving vehicles is a critical task for any autonomously operating outdoor robot or self-driving vehicle. Most modern approaches for solving this task rely on training image-based detectors using large-scale vehicle detection datasets such as nuScenes or the Waymo Open Dataset. Providing manual annotations is an expensive and laborious exercise that does not scale well in practice. To tackle this problem, we propose a self-supervised approach that leverages audio-visual cues to detect moving vehicles in videos. Our approach employs contrastive learning for localizing vehicles in images from corresponding pairs of images and recorded audio. In extensive experiments carried out with a real-world dataset, we demonstrate that our approach provides accurate detections of moving vehicles and does not require manual annotations. We furthermore show that our model can be used as a teacher to supervise an audio-only detection model. This student model is invariant to illumination changes and thus effectively bridges the domain gap inherent to models leveraging exclusively vision as the predominant modality.