# Three-class Overlapped Speech Detection using a Convolutional Recurrent Neural Network

arXiv: https://arxiv.org/abs/2104.02878 · HTML: https://mlsanity.com/paper/2104.02878/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 31 references that are in the map.
- First posted on arXiv 7 Apr 2021.

## Builds on (most cited first, 11 of 11)

- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [MUSAN: A Music, Speech, and Noise Corpus](https://mlsanity.com/paper/1510.08484/) (2015)
- [Spot the conversation: speaker diarisation in the wild](https://mlsanity.com/paper/2007.01216/) (2020)
- [The Third DIHARD Diarization Challenge](https://mlsanity.com/paper/2012.01477/) (2020)
- [The Second DIHARD Diarization Challenge: Dataset, task, and baselines](https://mlsanity.com/paper/1906.07839/) (2019)
- [Clova Baseline System for the VoxCeleb Speaker Recognition Challenge 2020](https://mlsanity.com/paper/2009.14153/) (2020)
- [Overlap-aware diarization: resegmentation using neural end-to-end overlapped speech detection](https://mlsanity.com/paper/1910.11646/) (2019)
- [Polyphonic Sound Event Detection and Localization using a Two-Stage Strategy](https://mlsanity.com/paper/1905.00268/) (2019)
- [Third DIHARD Challenge Evaluation Plan](https://mlsanity.com/paper/2006.05815/) (2020)
- [USTC-NELSLIP System Description for DIHARD-III Challenge](https://mlsanity.com/paper/2103.10661/) (2021)
- [Frame-based overlapping speech detection using Convolutional Neural Networks](https://mlsanity.com/paper/2001.09937/) (2020)

## Built on by (most cited first, 8 of 8)

- [Online Neural Diarization of Unlimited Numbers of Speakers Using Global and Local Attractors](https://mlsanity.com/paper/2206.02432/) (2022)
- [Multi-scale speaker embedding-based graph attention networks for speaker diarisation](https://mlsanity.com/paper/2110.03361/) (2021)
- [Overlapped speech and gender detection with WavLM pre-trained features](https://mlsanity.com/paper/2209.04167/) (2022)
- [End-to-End Integration of Speech Separation and Voice Activity Detection for Low-Latency Diarization of Telephone Conversations](https://mlsanity.com/paper/2303.12002/) (2023)
- [Joint speech and overlap detection: a benchmark over multiple audio setup and speech domains](https://mlsanity.com/paper/2307.13012/) (2023)
- [An Explainable Proxy Model for Multiabel Audio Segmentation](https://mlsanity.com/paper/2401.08268/) (2024)
- [Towards Robust Overlapping Speech Detection: A Speaker-Aware Progressive Approach Using WavLM](https://mlsanity.com/paper/2505.23207/) (2025)
- [Explainable by-design Audio Segmentation through Non-Negative Matrix Factorization and Probing](https://mlsanity.com/paper/2406.13385/) (2024)

## Abstract (arXiv)

In this work, we propose an overlapped speech detection system trained as a three-class classifier. Unlike conventional systems that perform binary classification as to whether or not a frame contains overlapped speech, the proposed approach classifies into three classes: non-speech, single speaker speech, and overlapped speech. By training a network with the more detailed label definition, the model can learn a better notion on deciding the number of speakers included in a given frame. A convolutional recurrent neural network architecture is explored to benefit from both convolutional layer's capability to model local patterns and recurrent layer's ability to model sequential information. The proposed overlapped speech detection model establishes a state-of-the-art performance with a precision of 0.6648 and a recall of 0.3222 on the DIHARD II evaluation set, showing a 20% increase in recall along with higher precision. In addition, we also introduce a simple approach to utilize the proposed overlapped speech detection model for speaker diarization which ranked third place in the Track 1 of the DIHARD III challenge.