# Self-supervised Pretraining for Robust Personalized Voice Activity Detection in Adverse Conditions

arXiv: https://arxiv.org/abs/2312.16613 · HTML: https://mlsanity.com/paper/2312.16613/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 24 references that are in the map.
- First posted on arXiv 27 Dec 2023.

## Builds on (most cited first, 16 of 16)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Libri-Light: A Benchmark for ASR with Limited or No Supervision](https://mlsanity.com/paper/1912.07875/) (2019)
- [Generalized End-to-End Loss for Speaker Verification](https://mlsanity.com/paper/1710.10467/) (2017)
- [An Unsupervised Autoregressive Model for Speech Representation Learning](https://mlsanity.com/paper/1904.03240/) (2019)
- [Self-supervised Learning is More Robust to Dataset Imbalance](https://mlsanity.com/paper/2110.05025/) (2021)
- [Generative Pre-Training for Speech with Autoregressive Predictive Coding](https://mlsanity.com/paper/1910.12607/) (2019)
- [Target-Speaker Voice Activity Detection: a Novel Approach for Multi-Speaker Diarization in a Dinner Party Scenario](https://mlsanity.com/paper/2005.07272/) (2020)
- [rVAD: An Unsupervised Segment-Based Robust Voice Activity Detection Method](https://mlsanity.com/paper/1906.03588/) (2019)
- [Personal VAD: Speaker-Conditioned Voice Activity Detection](https://mlsanity.com/paper/1908.04284/) (2019)
- [Target-speaker Voice Activity Detection with Improved I-Vector Estimation for Unknown Number of Speaker](https://mlsanity.com/paper/2108.03342/) (2021)
- [Personal VAD 2.0: Optimizing Personal Voice Activity Detection for On-Device Speech Recognition](https://mlsanity.com/paper/2204.03793/) (2022)
- [Voice activity detection in the wild: A data-driven approach using teacher-student training](https://mlsanity.com/paper/2105.04065/) (2021)
- [Understanding The Robustness of Self-supervised Learning Through Topic Modeling](https://mlsanity.com/paper/2203.03539/) (2022)
- [Cross-domain Voice Activity Detection with Self-Supervised Representations](https://mlsanity.com/paper/2209.11061/) (2022)

## Built on by (most cited first, 3 of 3)

- [Investigation of Speaker Representation for Target-Speaker Speech Processing](https://mlsanity.com/paper/2410.11243/) (2024)
- [Noise-Robust Target-Speaker Voice Activity Detection Through Self-Supervised Pretraining](https://mlsanity.com/paper/2501.03184/) (2025)
- [LibriVAD: A Scalable Open Dataset with Deep Learning Benchmarks for Voice Activity Detection](https://mlsanity.com/paper/2512.17281/) (2025)

## Abstract (arXiv)

In this paper, we propose the use of self-supervised pretraining on a large unlabelled data set to improve the performance of a personalized voice activity detection (VAD) model in adverse conditions. We pretrain a long short-term memory (LSTM)-encoder using the autoregressive predictive coding (APC) framework and fine-tune it for personalized VAD. We also propose a denoising variant of APC, with the goal of improving the robustness of personalized VAD. The trained models are systematically evaluated on both clean speech and speech contaminated by various types of noise at different SNR-levels and compared to a purely supervised model. Our experiments show that self-supervised pretraining not only improves performance in clean conditions, but also yields models which are more robust to adverse conditions compared to purely supervised learning.