# Exploring Self-Supervised Contrastive Learning of Spatial Sound Event Representation

arXiv: https://arxiv.org/abs/2309.15938 · HTML: https://mlsanity.com/paper/2309.15938/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 31 references that are in the map.
- First posted on arXiv 27 Sep 2023.

## Builds on (most cited first, 24 of 24)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [BEATs: Audio Pre-Training with Acoustic Tokenizers](https://mlsanity.com/paper/2212.09058/) (2022)
- [FSD50K: An Open Dataset of Human-Labeled Sound Events](https://mlsanity.com/paper/2010.00475/) (2020)
- [Masked Autoencoders that Listen](https://mlsanity.com/paper/2207.06405/) (2022)
- [Contrastive Learning of General-Purpose Audio Representations](https://mlsanity.com/paper/2010.10915/) (2020)
- [SSAST: Self-Supervised Audio Spectrogram Transformer](https://mlsanity.com/paper/2110.09784/) (2021)
- [BYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representation](https://mlsanity.com/paper/2103.06695/) (2021)
- [Sound Event Localization and Detection of Overlapping Sources Using Convolutional Recurrent Neural Networks](https://mlsanity.com/paper/1807.00129/) (2018)
- [Contrastive Learning of Musical Representations](https://mlsanity.com/paper/2103.09410/) (2021)
- [gpuRIR: A Python Library for Room Impulse Response Simulation with GPU Acceleration](https://mlsanity.com/paper/1810.11359/) (2018)
- [Multi-Format Contrastive Learning of Audio Representations](https://mlsanity.com/paper/2103.06508/) (2021)
- [Unsupervised Contrastive Learning of Sound Event Representations](https://mlsanity.com/paper/2011.07616/) (2020)
- [Overview and Evaluation of Sound Event Localization and Detection in DCASE 2019](https://mlsanity.com/paper/2009.02792/) (2020)
- [Learning Sound Event Classifiers from Web Audio with Noisy Labels](https://mlsanity.com/paper/1901.01189/) (2019)
- [A Four-Stage Data Augmentation Approach to ResNet-Conformer Based Acoustic Modeling for Sound Event Localization and Detection](https://mlsanity.com/paper/2101.02919/) (2021)
- [Polyphonic Sound Event Detection and Localization using a Two-Stage Strategy](https://mlsanity.com/paper/1905.00268/) (2019)
- [Sound Localization by Self-Supervised Time Delay Estimation](https://mlsanity.com/paper/2204.12489/) (2022)
- [SALSA-Lite: A Fast and Effective Feature for Polyphonic Sound Event Localization and Detection with Microphone Arrays](https://mlsanity.com/paper/2111.08192/) (2021)
- [Assessment of Self-Attention on Learned Features For Sound Event Localization and Detection](https://mlsanity.com/paper/2107.09388/) (2021)
- [Spatial Data Augmentation with Simulated Room Impulse Responses for Sound Event Localization and Detection](https://mlsanity.com/paper/2110.06501/) (2021)

## Built on by (most cited first, 2 of 2)

- [Sci-Phi: A Large Language Model Spatial Audio Descriptor](https://mlsanity.com/paper/2510.05542/) (2025)
- [Feature Aggregation in Joint Sound Classification and Localization Neural Networks](https://mlsanity.com/paper/2310.19063/) (2023)

## Abstract (arXiv)

In this study, we present a simple multi-channel framework for contrastive learning (MC-SimCLR) to encode 'what' and 'where' of spatial audios. MC-SimCLR learns joint spectral and spatial representations from unlabeled spatial audios, thereby enhancing both event classification and sound localization in downstream tasks. At its core, we propose a multi-level data augmentation pipeline that augments different levels of audio features, including waveforms, Mel spectrograms, and generalized cross-correlation (GCC) features. In addition, we introduce simple yet effective channel-wise augmentation methods to randomly swap the order of the microphones and mask Mel and GCC channels. By using these augmentations, we find that linear layers on top of the learned representation significantly outperform supervised models in terms of both event classification accuracy and localization error. We also perform a comprehensive analysis of the effect of each augmentation method and a comparison of the fine-tuning performance using different amounts of labeled data.