# Overlapped speech and gender detection with WavLM pre-trained features

arXiv: https://arxiv.org/abs/2209.04167 · HTML: https://mlsanity.com/paper/2209.04167/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 27 references that are in the map.
- First posted on arXiv 9 Sep 2022.

## Builds on (most cited first, 13 of 13)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [pyannote.audio: neural building blocks for speaker diarization](https://mlsanity.com/paper/1911.01255/) (2019)
- [Speaker Recognition from Raw Waveform with SincNet](https://mlsanity.com/paper/1808.00158/) (2018)
- [End-to-end speaker segmentation for overlap-aware resegmentation](https://mlsanity.com/paper/2104.04045/) (2021)
- [The Third DIHARD Diarization Challenge](https://mlsanity.com/paper/2012.01477/) (2020)
- [Overlap-aware diarization: resegmentation using neural end-to-end overlapped speech detection](https://mlsanity.com/paper/1910.11646/) (2019)
- [Three-class Overlapped Speech Detection using a Convolutional Recurrent Neural Network](https://mlsanity.com/paper/2104.02878/) (2021)
- [Speaker detection in the wild: Lessons learned from JSALT 2019](https://mlsanity.com/paper/1912.00938/) (2019)

## Built on by (most cited first, 6 of 6)

- [Bridge to Non-Barrier Communication: Gloss-Prompted Fine-grained Cued Speech Gesture Generation with Diffusion Model](https://mlsanity.com/paper/2404.19277/) (2024)
- [Joint speech and overlap detection: a benchmark over multiple audio setup and speech domains](https://mlsanity.com/paper/2307.13012/) (2023)
- [An Explainable Proxy Model for Multiabel Audio Segmentation](https://mlsanity.com/paper/2401.08268/) (2024)
- [ASoBO: Attentive Beamformer Selection for Distant Speaker Diarization in Meetings](https://mlsanity.com/paper/2406.03251/) (2024)
- [Beyond Neural-on-Neural Approaches to Speaker Gender Protection](https://mlsanity.com/paper/2306.17700/) (2023)
- [Explainable by-design Audio Segmentation through Non-Negative Matrix Factorization and Probing](https://mlsanity.com/paper/2406.13385/) (2024)

## Abstract (arXiv)

This article focuses on overlapped speech and gender detection in order to study interactions between women and men in French audiovisual media (Gender Equality Monitoring project). In this application context, we need to automatically segment the speech signal according to speakers gender, and to identify when at least two speakers speak at the same time. We propose to use WavLM model which has the advantage of being pre-trained on a huge amount of speech data, to build an overlapped speech detection (OSD) and a gender detection (GD) systems. In this study, we use two different corpora. The DIHARD III corpus which is well adapted for the OSD task but lack gender information. The ALLIES corpus fits with the project application context. Our best OSD system is a Temporal Convolutional Network (TCN) with WavLM pre-trained features as input, which reaches a new state-of-the-art F1-score performance on DIHARD. A neural GD is trained with WavLM inputs on a gender balanced subset of the French broadcast news ALLIES data, and obtains an accuracy of 97.9%. This work opens new perspectives for human science researchers regarding the differences of representation between women and men in French media.