# Joint Speech Activity and Overlap Detection with Multi-Exit Architecture

arXiv: https://arxiv.org/abs/2209.11906 · HTML: https://mlsanity.com/paper/2209.11906/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 39 references that are in the map.
- First posted on arXiv 24 Sep 2022.

## Builds on (most cited first, 16 of 16)

- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks](https://mlsanity.com/paper/1709.01686/) (2017)
- [An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition](https://mlsanity.com/paper/1507.05717/) (2015)
- [MUSAN: A Music, Speech, and Noise Corpus](https://mlsanity.com/paper/1510.08484/) (2015)
- [Dynamic Neural Networks: A Survey](https://mlsanity.com/paper/2102.04906/) (2021)
- [pyannote.audio: neural building blocks for speaker diarization](https://mlsanity.com/paper/1911.01255/) (2019)
- [Speaker Recognition from Raw Waveform with SincNet](https://mlsanity.com/paper/1808.00158/) (2018)
- [CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordings](https://mlsanity.com/paper/2004.09249/) (2020)
- [End-to-end speaker segmentation for overlap-aware resegmentation](https://mlsanity.com/paper/2104.04045/) (2021)
- [Spot the conversation: speaker diarisation in the wild](https://mlsanity.com/paper/2007.01216/) (2020)
- [The Third DIHARD Diarization Challenge](https://mlsanity.com/paper/2012.01477/) (2020)
- [Overlap-aware diarization: resegmentation using neural end-to-end overlapped speech detection](https://mlsanity.com/paper/1910.11646/) (2019)
- [Multi-class Spectral Clustering with Overlaps for Speaker Diarization](https://mlsanity.com/paper/2011.02900/) (2020)
- [Frame-based overlapping speech detection using Convolutional Neural Networks](https://mlsanity.com/paper/2001.09937/) (2020)
- [End-to-End Speaker Diarization Conditioned on Speech Activity and Overlap Detection](https://mlsanity.com/paper/2106.04078/) (2021)
- [Students are the Best Teacher: Exit-Ensemble Distillation with Multi-Exits](https://mlsanity.com/paper/2104.00299/) (2021)

## Abstract (arXiv)

Overlapped speech detection (OSD) is critical for speech applications in scenario of multi-party conversion. Despite numerous research efforts and progresses, comparing with speech activity detection (VAD), OSD remains an open challenge and its overall performance is far from satisfactory. The majority of prior research typically formulates the OSD problem as a standard classification problem, to identify speech with binary (OSD) or three-class label (joint VAD and OSD) at frame level. In contrast to the mainstream, this study investigates the joint VAD and OSD task from a new perspective. In particular, we propose to extend traditional classification network with multi-exit architecture. Such an architecture empowers our system with unique capability to identify class using either low-level features from early exits or high-level features from last exit. In addition, two training schemes, knowledge distillation and dense connection, are adopted to further boost our system performance. Experimental results on benchmark datasets (AMI and DIHARD-III) validated the effectiveness and generality of our proposed system. Our ablations further reveal the complementary contribution of proposed schemes. With F1 score of 0.792 on AMI and 0.625 on DIHARD-III, our proposed system outperforms several top performing models on these datasets, but also surpasses the current state-of-the-art by large margins across both datasets. Besides the performance benefit, our proposed system offers another appealing potential for quality-complexity trade-offs, which is highly preferred for efficient OSD deployment.