# Towards Word-Level End-to-End Neural Speaker Diarization with Auxiliary Network

arXiv: https://arxiv.org/abs/2309.08489 · HTML: https://mlsanity.com/paper/2309.08489/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 42 references that are in the map.
- First posted on arXiv 15 Sep 2023.

## Builds on (most cited first, 29 of 29)

- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing](https://mlsanity.com/paper/2006.03236/) (2020)
- [A Review of Speaker Diarization: Recent Advances with Deep Learning](https://mlsanity.com/paper/2101.09624/) (2021)
- [Hybrid Autoregressive Transducer (hat)](https://mlsanity.com/paper/2003.07705/) (2020)
- [End-to-End Neural Speaker Diarization with Permutation-Free Objectives](https://mlsanity.com/paper/1909.05952/) (2019)
- [Speaker Diarization with LSTM](https://mlsanity.com/paper/1710.10468/) (2017)
- [End-to-End Speaker Diarization for an Unknown Number of Speakers with Encoder-Decoder Based Attractors](https://mlsanity.com/paper/2005.09921/) (2020)
- [Target-Speaker Voice Activity Detection: a Novel Approach for Multi-Speaker Diarization in a Dinner Party Scenario](https://mlsanity.com/paper/2005.07272/) (2020)
- [Joint Speech Recognition and Speaker Diarization via Sequence Transduction](https://mlsanity.com/paper/1907.05337/) (2019)
- [Fully Supervised Speaker Diarization](https://mlsanity.com/paper/1810.04719/) (2018)
- [ECAPA-TDNN Embeddings for Speaker Diarization](https://mlsanity.com/paper/2104.01466/) (2021)
- [Joint Speaker Counting, Speech Recognition, and Speaker Identification for Overlapped Speech of Any Number of Speakers](https://mlsanity.com/paper/2006.10930/) (2020)
- [Tied & Reduced RNN-T Decoder](https://mlsanity.com/paper/2109.07513/) (2021)
- [Integrating end-to-end neural and clustering-based diarization: Getting the best of both worlds](https://mlsanity.com/paper/2010.13366/) (2020)
- [End-to-End Speaker-Attributed ASR with Transformer](https://mlsanity.com/paper/2104.02128/) (2021)
- [Turn-to-Diarize: Online Speaker Diarization Constrained by Transformer Transducer Speaker Turn Detection](https://mlsanity.com/paper/2109.11641/) (2021)
- [Investigation of End-To-End Speaker-Attributed ASR for Continuous Multi-Talker Recordings](https://mlsanity.com/paper/2008.04546/) (2020)
- [Improving RNN Transducer Based ASR with Auxiliary Tasks](https://mlsanity.com/paper/2011.03109/) (2020)
- [Auxiliary Interference Speaker Loss for Target-Speaker Speech Recognition](https://mlsanity.com/paper/1906.10876/) (2019)
- [Discriminative Neural Clustering for Speaker Diarisation](https://mlsanity.com/paper/1910.09703/) (2019)
- [Neural Speaker Diarization with Speaker-Wise Chain Rule](https://mlsanity.com/paper/2006.01796/) (2020)
- [Online End-to-End Neural Diarization with Speaker-Tracing Buffer](https://mlsanity.com/paper/2006.02616/) (2020)
- [Streaming Speaker-Attributed ASR with Token-Level Speaker Embeddings](https://mlsanity.com/paper/2203.16685/) (2022)
- [E2E-based Multi-task Learning Approach to Joint Speech and Accent Recognition](https://mlsanity.com/paper/2106.08211/) (2021)
- [BW-EDA-EEND: Streaming End-to-End Neural Speaker Diarization for a Variable Number of Speakers](https://mlsanity.com/paper/2011.02678/) (2020)
- [Minimum Bayes Risk Training for End-to-End Speaker-Attributed ASR](https://mlsanity.com/paper/2011.02921/) (2020)
- [Highly Efficient Real-Time Streaming and Fully On-Device Speaker Diarization with Multi-Stage Clustering](https://mlsanity.com/paper/2210.13690/) (2022)
- [Augmenting Transformer-Transducer Based Speaker Change Detection With Token-Level Training Loss](https://mlsanity.com/paper/2211.06482/) (2022)
- [Intermediate-layer output Regularization for Attention-based Speech Recognition with Shared Decoder](https://mlsanity.com/paper/2207.04177/) (2022)

## Built on by (most cited first, 1 of 1)

- [DiarizationLM: Speaker Diarization Post-Processing with Large Language Models](https://mlsanity.com/paper/2401.03506/) (2024)

## Abstract (arXiv)

While standard speaker diarization attempts to answer the question "who spoken when", most of relevant applications in reality are more interested in determining "who spoken what". Whether it is the conventional modularized approach or the more recent end-to-end neural diarization (EEND), an additional automatic speech recognition (ASR) model and an orchestration algorithm are required to associate the speaker labels with recognized words. In this paper, we propose Word-level End-to-End Neural Diarization (WEEND) with auxiliary network, a multi-task learning algorithm that performs end-to-end ASR and speaker diarization in the same neural architecture. That is, while speech is being recognized, speaker labels are predicted simultaneously for each recognized word. Experimental results demonstrate that WEEND outperforms the turn-based diarization baseline system on all 2-speaker short-form scenarios and has the capability to generalize to audio lengths of 5 minutes. Although 3+speaker conversations are harder, we find that with enough in-domain training data, WEEND has the potential to deliver high quality diarized text.