# Semi-supervised source localization in reverberant environments with deep generative modeling

arXiv: https://arxiv.org/abs/2101.10636 · HTML: https://mlsanity.com/paper/2101.10636/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 74 references that are in the map.
- First posted on arXiv 26 Jan 2021.

## Builds on (most cited first, 23 of 23)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [An Introduction to Variational Autoencoders](https://mlsanity.com/paper/1906.02691/) (2019)
- [Semi-Supervised Learning with Deep Generative Models](https://mlsanity.com/paper/1406.5298/) (2014)
- [Pyro: Deep Universal Probabilistic Programming](https://mlsanity.com/paper/1810.09538/) (2018)
- [Pyroomacoustics: A Python package for audio room simulations and array processing algorithms](https://mlsanity.com/paper/1710.04196/) (2017)
- [Deep Learning for Audio Signal Processing](https://mlsanity.com/paper/1905.00078/) (2019)
- [Sound Event Localization and Detection of Overlapping Sources Using Convolutional Recurrent Neural Networks](https://mlsanity.com/paper/1807.00129/) (2018)
- [Learning Disentangled Representations with Semi-Supervised Deep Generative Models](https://mlsanity.com/paper/1706.00400/) (2017)
- [Machine learning in acoustics: theory and applications](https://mlsanity.com/paper/1905.04418/) (2019)
- [Exploring Speech Enhancement with Generative Adversarial Networks for Robust Speech Recognition](https://mlsanity.com/paper/1711.05747/) (2017)
- [LOCA: LOcal Conformal Autoencoder for standardized data coordinates](https://mlsanity.com/paper/2004.07234/) (2020)
- [The LOCATA Challenge: Acoustic Source Localization and Tracking](https://mlsanity.com/paper/1909.01008/) (2019)
- [Semi-supervised multichannel speech enhancement with variational autoencoders and non-negative matrix factorization](https://mlsanity.com/paper/1811.06713/) (2018)
- [Multi-Speaker DOA Estimation Using Deep Convolutional Networks Trained with Noise Signals](https://mlsanity.com/paper/1807.11722/) (2018)
- [Semi-supervised source localization with deep generative modeling](https://mlsanity.com/paper/2005.13163/) (2020)
- [Semi-Supervised Sound Source Localization Based on Manifold Regularization](https://mlsanity.com/paper/1508.03148/) (2015)
- [Spatial Source Subtraction Based on Incomplete Measurements of Relative Transfer Function](https://mlsanity.com/paper/1411.2744/) (2014)
- [Feature Selection based on Principal Component Analysis for Underwater Source Localization by Deep Learning](https://mlsanity.com/paper/2011.12754/) (2020)

## Built on by (most cited first, 5 of 5)

- [A Survey of Sound Source Localization with Deep Learning Methods](https://mlsanity.com/paper/2109.03465/) (2021)
- [Generative adversarial networks with physical sound field priors](https://mlsanity.com/paper/2308.00426/) (2023)
- [Multi-modal Iterative and Deep Fusion Frameworks for Enhanced Passive DOA Sensing via a Green Massive H2AD MIMO Receiver](https://mlsanity.com/paper/2411.06927/) (2024)
- [Audio scene monitoring using redundant ad-hoc microphone array networks](https://mlsanity.com/paper/2103.01830/) (2021)
- [Feature Aggregation in Joint Sound Classification and Localization Neural Networks](https://mlsanity.com/paper/2310.19063/) (2023)

## Abstract (arXiv)

We propose a semi-supervised approach to acoustic source localization in reverberant environments based on deep generative modeling. Localization in reverberant environments remains an open challenge. Even with large data volumes, the number of labels available for supervised learning in reverberant environments is usually small. We address this issue by performing semi-supervised learning (SSL) with convolutional variational autoencoders (VAEs) on reverberant speech signals recorded with microphone arrays. The VAE is trained to generate the phase of relative transfer functions (RTFs) between microphones, in parallel with a direction of arrival (DOA) classifier based on RTF-phase. These models are trained using both labeled and unlabeled RTF-phase sequences. In learning to perform these tasks, the VAE-SSL explicitly learns to separate the physical causes of the RTF-phase (i.e., source location) from distracting signal characteristics such as noise and speech activity. Relative to existing semi-supervised localization methods in acoustics, VAE-SSL is effectively an end-to-end processing approach which relies on minimal preprocessing of RTF-phase features. As far as we are aware, our paper presents the first approach to modeling the physics of acoustic propagation using deep generative modeling. The VAE-SSL approach is compared with two signal processing-based approaches, steered response power with phase transform (SRP-PHAT) and MUltiple SIgnal Classification (MUSIC), as well as fully supervised CNNs. We find that VAE-SSL can outperform the conventional approaches and the CNN in label-limited scenarios. Further, the trained VAE-SSL system can generate new RTF-phase samples, which shows the VAE-SSL approach learns the physics of the acoustic environment. The generative modeling in VAE-SSL thus provides a means of interpreting the learned representations.