# Enhanced Sound Event Localization and Detection in Real 360-degree audio-visual soundscapes

arXiv: https://arxiv.org/abs/2401.17129 · HTML: https://mlsanity.com/paper/2401.17129/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 26 references that are in the map.
- First posted on arXiv 29 Jan 2024.

## Builds on (most cited first, 21 of 21)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [YOLOX: Exceeding YOLO Series in 2021](https://mlsanity.com/paper/2107.08430/) (2021)
- [VGGSound: A Large-scale Audio-Visual Dataset](https://mlsanity.com/paper/2004.14368/) (2020)
- [Detecting Twenty-thousand Classes using Image-level Supervision](https://mlsanity.com/paper/2201.02605/) (2022)
- [FSD50K: An Open Dataset of Human-Labeled Sound Events](https://mlsanity.com/paper/2010.00475/) (2020)
- [Sound Event Localization and Detection of Overlapping Sources Using Convolutional Recurrent Neural Networks](https://mlsanity.com/paper/1807.00129/) (2018)
- [Localizing Visual Sounds the Easy Way](https://mlsanity.com/paper/2203.09324/) (2022)
- [A Closer Look at Weakly-Supervised Audio-Visual Source Localization](https://mlsanity.com/paper/2209.09634/) (2022)
- [A Survey of Sound Source Localization with Deep Learning Methods](https://mlsanity.com/paper/2109.03465/) (2021)
- [STARSS23: An Audio-Visual Dataset of Spatial Recordings of Real Scenes with Spatiotemporal Annotations of Sound Events](https://mlsanity.com/paper/2306.09126/) (2023)
- [Overview and Evaluation of Sound Event Localization and Detection in DCASE 2019](https://mlsanity.com/paper/2009.02792/) (2020)
- [Multi-ACCDOA: Localizing and Detecting Overlapping Sounds from the Same Class with Auxiliary Duplicating Permutation Invariant Training](https://mlsanity.com/paper/2110.07124/) (2021)
- [A Four-Stage Data Augmentation Approach to ResNet-Conformer Based Acoustic Modeling for Sound Event Localization and Detection](https://mlsanity.com/paper/2101.02919/) (2021)
- [A multi-room reverberant dataset for sound event localization and detection](https://mlsanity.com/paper/1905.08546/) (2019)
- [Spatial Scaper: A Library to Simulate and Augment Soundscapes for Sound Event Localization and Detection in Realistic Rooms](https://mlsanity.com/paper/2401.12238/) (2024)
- [Localization, Detection and Tracking of Multiple Moving Sound Sources with a Convolutional Recurrent Neural Network](https://mlsanity.com/paper/1904.12769/) (2019)
- [Sound Source Distance Estimation in Diverse and Dynamic Acoustic Conditions](https://mlsanity.com/paper/2309.09288/) (2023)
- [Assessment of Self-Attention on Learned Features For Sound Event Localization and Detection](https://mlsanity.com/paper/2107.09388/) (2021)
- [Robust DOA estimation using deep acoustic imaging](https://mlsanity.com/paper/2401.08717/) (2024)
- [Bridging High-Quality Audio and Video via Language for Sound Effects Retrieval from Visual Queries](https://mlsanity.com/paper/2308.09089/) (2023)

## Built on by (most cited first, 3 of 3)

- [Leveraging Reverberation and Visual Depth Cues for Sound Event Localization and Detection with Distance Estimation](https://mlsanity.com/paper/2410.22271/) (2024)
- [Spatial and Semantic Embedding Integration for Stereo Sound Event Localization and Detection in Regular Videos](https://mlsanity.com/paper/2507.04845/) (2025)
- [Integrating Spatial and Semantic Embeddings for Stereo Sound Event Localization in Videos](https://mlsanity.com/paper/2509.06598/) (2025)

## Abstract (arXiv)

This technical report details our work towards building an enhanced audio-visual sound event localization and detection (SELD) network. We build on top of the audio-only SELDnet23 model and adapt it to be audio-visual by merging both audio and video information prior to the gated recurrent unit (GRU) of the audio-only network. Our model leverages YOLO and DETIC object detectors. We also build a framework that implements audio-visual data augmentation and audio-visual synthetic data generation. We deliver an audio-visual SELDnet system that outperforms the existing audio-visual SELD baseline.