mlsanityData to 7 Oct 2026

Enhanced Sound Event Localization and Detection in Real 360-degree audio-visual soundscapes

arXiv 2401.17129 ↗First posted 29 Jan 2024cs.SD cs.AI eess.AScited by 3

Not on the map. Why?

Abstract (arXiv)

This technical report details our work towards building an enhanced audio-visual sound event localization and detection (SELD) network. We build on top of the audio-only SELDnet23 model and adapt it to be audio-visual by merging both audio and video information prior to the gated recurrent unit (GRU) of the audio-only network. Our model leverages YOLO and DETIC object detectors. We also build a framework that implements audio-visual data augmentation and audio-visual synthetic data generation. We deliver an audio-visual SELDnet system that outperforms the existing audio-visual SELD baseline.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 21 of its 26 references that are in the map. First posted on arXiv 29 Jan 2024.

Builds on

21 of 26 references.

References · 21

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  3. 2021YOLOX: Exceeding YOLO Series in 2021cited by 1,064
  4. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  5. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  6. 2020FSD50K: An Open Dataset of Human-Labeled Sound Eventscited by 339
  7. 2018Sound Event Localization and Detection of Overlapping Sources Using Convolutional Recurrent Neural Networkscited by 84
  8. 2022Localizing Visual Sounds the Easy Waycited by 81
  9. 2022A Closer Look at Weakly-Supervised Audio-Visual Source Localizationcited by 72
  10. 2021A Survey of Sound Source Localization with Deep Learning Methodscited by 45
  11. 2023STARSS23: An Audio-Visual Dataset of Spatial Recordings of Real Scenes with Spatiotemporal Annotations of Sound Eventscited by 39
  12. 2020Overview and Evaluation of Sound Event Localization and Detection in DCASE 2019cited by 30
9 more
  1. 2021Multi-ACCDOA: Localizing and Detecting Overlapping Sounds from the Same Class with Auxiliary Duplicating Permutation Invariant Trainingcited by 23
  2. 2021A Four-Stage Data Augmentation Approach to ResNet-Conformer Based Acoustic Modeling for Sound Event Localization and Detectioncited by 21
  3. 2019A multi-room reverberant dataset for sound event localization and detectioncited by 18
  4. 2024Spatial Scaper: A Library to Simulate and Augment Soundscapes for Sound Event Localization and Detection in Realistic Roomscited by 12
  5. 2019Localization, Detection and Tracking of Multiple Moving Sound Sources with a Convolutional Recurrent Neural Networkcited by 9
  6. 2023Sound Source Distance Estimation in Diverse and Dynamic Acoustic Conditionscited by 8
  7. 2021Assessment of Self-Attention on Learned Features For Sound Event Localization and Detectioncited by 7
  8. 2024Robust DOA estimation using deep acoustic imagingcited by 4
  9. 2023Bridging High-Quality Audio and Video via Language for Sound Effects Retrieval from Visual Queriescited by 2

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 2.

3 papers.

  1. 2024Leveraging Reverberation and Visual Depth Cues for Sound Event Localization and Detection with Distance Estimationcited by 3
  2. 2025Spatial and Semantic Embedding Integration for Stereo Sound Event Localization and Detection in Regular Videoscited by 1
  3. 2025Integrating Spatial and Semantic Embeddings for Stereo Sound Event Localization in Videoscited by 1