# Joint speech and overlap detection: a benchmark over multiple audio setup and speech domains

arXiv: https://arxiv.org/abs/2307.13012 · HTML: https://mlsanity.com/paper/2307.13012/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 29 references that are in the map.
- First posted on arXiv 24 Jul 2023.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [MUSAN: A Music, Speech, and Noise Corpus](https://mlsanity.com/paper/1510.08484/) (2015)
- [pyannote.audio: neural building blocks for speaker diarization](https://mlsanity.com/paper/1911.01255/) (2019)
- [The fifth 'CHiME' Speech Separation and Recognition Challenge: Dataset, task and baselines](https://mlsanity.com/paper/1803.10609/) (2018)
- [End-to-end speaker segmentation for overlap-aware resegmentation](https://mlsanity.com/paper/2104.04045/) (2021)
- [M2MeT: The ICASSP 2022 Multi-Channel Multi-Party Meeting Transcription Challenge](https://mlsanity.com/paper/2110.07393/) (2021)
- [The Second DIHARD Diarization Challenge: Dataset, task, and baselines](https://mlsanity.com/paper/1906.07839/) (2019)
- [Overlap-aware diarization: resegmentation using neural end-to-end overlapped speech detection](https://mlsanity.com/paper/1910.11646/) (2019)
- [End-to-end Domain-Adversarial Voice Activity Detection](https://mlsanity.com/paper/1910.10655/) (2019)
- [BUT System for the Second DIHARD Speech Diarization Challenge](https://mlsanity.com/paper/2002.11356/) (2020)
- [Three-class Overlapped Speech Detection using a Convolutional Recurrent Neural Network](https://mlsanity.com/paper/2104.02878/) (2021)
- [Speaker detection in the wild: Lessons learned from JSALT 2019](https://mlsanity.com/paper/1912.00938/) (2019)
- [Overlapped speech and gender detection with WavLM pre-trained features](https://mlsanity.com/paper/2209.04167/) (2022)
- [Self-Attention Channel Combinator Frontend for End-to-End Multichannel Far-field Speech Recognition](https://mlsanity.com/paper/2109.04783/) (2021)

## Built on by (most cited first, 2 of 2)

- [An Explainable Proxy Model for Multiabel Audio Segmentation](https://mlsanity.com/paper/2401.08268/) (2024)
- [Towards Robust Overlapping Speech Detection: A Speaker-Aware Progressive Approach Using WavLM](https://mlsanity.com/paper/2505.23207/) (2025)

## Abstract (arXiv)

Voice activity and overlapped speech detection (respectively VAD and OSD) are key pre-processing tasks for speaker diarization. The final segmentation performance highly relies on the robustness of these sub-tasks. Recent studies have shown VAD and OSD can be trained jointly using a multi-class classification model. However, these works are often restricted to a specific speech domain, lacking information about the generalization capacities of the systems. This paper proposes a complete and new benchmark of different VAD and OSD models, on multiple audio setups (single/multi-channel) and speech domains (e.g. media, meeting...). Our 2/3-class systems, which combine a Temporal Convolutional Network with speech representations adapted to the setup, outperform state-of-the-art results. We show that the joint training of these two tasks offers similar performances in terms of F1-score to two dedicated VAD and OSD systems while reducing the training cost. This unique architecture can also be used for single and multichannel speech processing.