# Fusion of Audio and Visual Embeddings for Sound Event Localization and Detection

arXiv: https://arxiv.org/abs/2312.09034 · HTML: https://mlsanity.com/paper/2312.09034/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Dec 2023.

## Built on by (most cited first, 9 of 9)

- [From Computation to Consumption: Exploring the Compute-Energy Link for Training and Testing Neural Networks for SED Systems](https://mlsanity.com/paper/2409.05080/) (2024)
- [Stereo Sound Event Localization and Detection with Onscreen/offscreen Classification](https://mlsanity.com/paper/2507.12042/) (2025)
- [Leveraging Reverberation and Visual Depth Cues for Sound Event Localization and Detection with Distance Estimation](https://mlsanity.com/paper/2410.22271/) (2024)
- [Quality Over Quantity? LLM-Based Curation for a Data-Efficient Audio-Video Foundation Model](https://mlsanity.com/paper/2503.09205/) (2025)
- [Semantic Audio-Visual Navigation in Continuous Environments](https://mlsanity.com/paper/2603.19660/) (2026)
- [Spatial and Semantic Embedding Integration for Stereo Sound Event Localization and Detection in Regular Videos](https://mlsanity.com/paper/2507.04845/) (2025)
- [Integrating Spatial and Semantic Embeddings for Stereo Sound Event Localization in Videos](https://mlsanity.com/paper/2509.06598/) (2025)
- [Audio-Visual Talker Localization in Video for Spatial Sound Reproduction](https://mlsanity.com/paper/2406.00495/) (2024)
- [ToS: A Team of Specialists ensemble framework for Stereo Sound Event Localization and Detection with distance estimation in Video](https://mlsanity.com/paper/2601.17611/) (2026)

## Abstract (arXiv)

Sound event localization and detection (SELD) combines two subtasks: sound event detection (SED) and direction of arrival (DOA) estimation. SELD is usually tackled as an audio-only problem, but visual information has been recently included. Few audio-visual (AV)-SELD works have been published and most employ vision via face/object bounding boxes, or human pose keypoints. In contrast, we explore the integration of audio and visual feature embeddings extracted with pre-trained deep networks. For the visual modality, we tested ResNet50 and Inflated 3D ConvNet (I3D). Our comparison of AV fusion methods includes the AV-Conformer and Cross-Modal Attentive Fusion (CMAF) model. Our best models outperform the DCASE 2023 Task3 audio-only and AV baselines by a wide margin on the development set of the STARSS23 dataset, making them competitive amongst state-of-the-art results of the AV challenge, without model ensembling, heavy data augmentation, or prediction post-processing. Such techniques and further pre-training could be applied as next steps to improve performance.