# Spatial and Semantic Embedding Integration for Stereo Sound Event Localization and Detection in Regular Videos arXiv: https://arxiv.org/abs/2507.04845 ยท HTML: https://mlsanity.com/paper/2507.04845/ ## Key facts - Cited by 1 papers in the map (data through 14 Oct 2026). - Builds on 33 of its 43 references that are in the map. - First posted on arXiv 7 Jul 2025. ## Builds on (most cited first, 30 of 33) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015) - [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020) - [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015) - [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022) - [FSD50K: An Open Dataset of Human-Labeled Sound Events](https://mlsanity.com/paper/2010.00475/) (2020) - [AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head](https://mlsanity.com/paper/2304.12995/) (2023) - [FMA: A Dataset For Music Analysis](https://mlsanity.com/paper/1612.01840/) (2016) - [Sound Event Localization and Detection of Overlapping Sources Using Convolutional Recurrent Neural Networks](https://mlsanity.com/paper/1807.00129/) (2018) - [STARSS23: An Audio-Visual Dataset of Spatial Recordings of Real Scenes with Spatiotemporal Annotations of Sound Events](https://mlsanity.com/paper/2306.09126/) (2023) - [Direction of arrival estimation for multiple sound sources using convolutional recurrent neural network](https://mlsanity.com/paper/1710.10059/) (2017) - [360-Indoor: Towards Learning Real-World Objects in 360deg Indoor Equirectangular Images](https://mlsanity.com/paper/1910.01712/) (2019) - [An Improved Event-Independent Network for Polyphonic Sound Event Localization and Detection](https://mlsanity.com/paper/2010.13092/) (2020) - [Multi-ACCDOA: Localizing and Detecting Overlapping Sounds from the Same Class with Auxiliary Duplicating Permutation Invariant Training](https://mlsanity.com/paper/2110.07124/) (2021) - [A Dataset of Reverberant Spatial Sound Scenes with Moving Sources for Sound Event Localization and Detection](https://mlsanity.com/paper/2006.01919/) (2020) - [A Four-Stage Data Augmentation Approach to ResNet-Conformer Based Acoustic Modeling for Sound Event Localization and Detection](https://mlsanity.com/paper/2101.02919/) (2021) - [A Dataset of Dynamic Reverberant Sound Scenes with Directional Interferers for Sound Event Localization and Detection](https://mlsanity.com/paper/2106.06999/) (2021) - [NitroFusion: High-Fidelity Single-Step Diffusion through Dynamic Adversarial Training](https://mlsanity.com/paper/2412.02030/) (2024) - [Sound Event Detection Using Spatial Features and Convolutional Recurrent Neural Network](https://mlsanity.com/paper/1706.02291/) (2017) - [Sound Event Detection and Localization with Distance Estimation](https://mlsanity.com/paper/2403.11827/) (2024) - [Robust and Adaptive Door Operation with a Mobile Robot](https://mlsanity.com/paper/1902.09051/) (2019) - [Spatial Scaper: A Library to Simulate and Augment Soundscapes for Sound Event Localization and Detection in Realistic Rooms](https://mlsanity.com/paper/2401.12238/) (2024) - [Event-Independent Network for Polyphonic Sound Event Localization and Detection](https://mlsanity.com/paper/2010.00140/) (2020) - [Dataset of Spatial Room Impulse Responses in a Variable Acoustics Room for Six Degrees-of-Freedom Rendering and Analysis](https://mlsanity.com/paper/2111.11882/) (2021) - [Fusion of Audio and Visual Embeddings for Sound Event Localization and Detection](https://mlsanity.com/paper/2312.09034/) (2023) - [Generating Diverse Audio-Visual 360 Soundscapes for Sound Event Localization and Detection](https://mlsanity.com/paper/2504.02988/) (2025) - [Exploring Audio-Visual Information Fusion for Sound Event Localization and Detection In Low-Resource Realistic Scenarios](https://mlsanity.com/paper/2406.15160/) (2024) - [Enhanced Sound Event Localization and Detection in Real 360-degree audio-visual soundscapes](https://mlsanity.com/paper/2401.17129/) (2024) ## Built on by (most cited first, 1 of 1) - [ToS: A Team of Specialists ensemble framework for Stereo Sound Event Localization and Detection with distance estimation in Video](https://mlsanity.com/paper/2601.17611/) (2026) ## Abstract (arXiv) This report presents our systems submitted to the audio-only and audio-visual tracks of the DCASE2025 Task 3 Challenge: Stereo Sound Event Localization and Detection (SELD) in Regular Video Content. SELD is a complex task that combines temporal event classification with spatial localization, requiring reasoning across spatial, temporal, and semantic dimensions. The last is arguably the most challenging to model. Traditional SELD architectures rely on multichannel input, which limits their ability to leverage large-scale pre-training due to data constraints. To address this, we enhance standard SELD architectures with semantic information by integrating pre-trained, contrastive language-aligned models: CLAP for audio and OWL-ViT for visual inputs. These embeddings are incorporated into a modified Conformer module tailored for multimodal fusion, which we refer to as the Cross-Modal Conformer. Additionally, we incorporate autocorrelation-based acoustic features to improve distance estimation. We pre-train our models on curated synthetic audio and audio-visual datasets and apply a left-right channel swapping augmentation to further increase the training data. Both our audio-only and audio-visual systems substantially outperform the challenge baselines on the development set, demonstrating the effectiveness of our strategy. Performance is further improved through model ensembling and a visual post-processing step based on human keypoints. Future work will investigate the contribution of each modality and explore architectural variants to further enhance results.