# Speaker Diarization of Scripted Audiovisual Content

arXiv: https://arxiv.org/abs/2308.02160 · HTML: https://mlsanity.com/paper/2308.02160/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 37 references that are in the map.
- First posted on arXiv 4 Aug 2023.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [In defence of metric learning for speaker recognition](https://mlsanity.com/paper/2003.11982/) (2020)
- [A Review of Speaker Diarization: Recent Advances with Deep Learning](https://mlsanity.com/paper/2101.09624/) (2021)
- [End-to-end speaker segmentation for overlap-aware resegmentation](https://mlsanity.com/paper/2104.04045/) (2021)
- [End-to-End Neural Speaker Diarization with Self-attention](https://mlsanity.com/paper/1909.06247/) (2019)
- [Speaker Diarization with LSTM](https://mlsanity.com/paper/1710.10468/) (2017)
- [End-to-End Speaker Diarization for an Unknown Number of Speakers with Encoder-Decoder Based Attractors](https://mlsanity.com/paper/2005.09921/) (2020)
- [Target-Speaker Voice Activity Detection: a Novel Approach for Multi-Speaker Diarization in a Dinner Party Scenario](https://mlsanity.com/paper/2005.07272/) (2020)
- [Fully Supervised Speaker Diarization](https://mlsanity.com/paper/1810.04719/) (2018)
- [ECAPA-TDNN Embeddings for Speaker Diarization](https://mlsanity.com/paper/2104.01466/) (2021)
- [Turn-to-Diarize: Online Speaker Diarization Constrained by Transformer Transducer Speaker Turn Detection](https://mlsanity.com/paper/2109.11641/) (2021)
- [Transcribe-to-Diarize: Neural Speaker Diarization for Unlimited Number of Speakers using End-to-End Speaker-Attributed ASR](https://mlsanity.com/paper/2110.03151/) (2021)
- [Dubbing in Practice: A Large Scale Study of Human Localization With Insights for Automatic Dubbing](https://mlsanity.com/paper/2212.12137/) (2022)
- [Speaker Diarization with Lexical Information](https://mlsanity.com/paper/2004.06756/) (2020)
- [Lexical Speaker Error Correction: Leveraging Language Models for Speaker Diarization Error Correction](https://mlsanity.com/paper/2306.09313/) (2023)
- [Exploiting Sentence Order in Document Alignment](https://mlsanity.com/paper/2004.14523/) (2020)
- [BW-EDA-EEND: Streaming End-to-End Neural Speaker Diarization for a Variable Number of Speakers](https://mlsanity.com/paper/2011.02678/) (2020)
- [Using Active Speaker Faces for Diarization in TV shows](https://mlsanity.com/paper/2203.15961/) (2022)
- [Audiovisual speaker diarization of TV series](https://mlsanity.com/paper/1812.07205/) (2018)
- [Constrained speaker diarization of TV series based on visual patterns](https://mlsanity.com/paper/1812.07209/) (2018)

## Built on by (most cited first, 1 of 1)

- [End-to-End Single-Channel Speaker-Turn Aware Conversational Speech Translation](https://mlsanity.com/paper/2311.00697/) (2023)

## Abstract (arXiv)

The media localization industry usually requires a verbatim script of the final film or TV production in order to create subtitles or dubbing scripts in a foreign language. In particular, the verbatim script (i.e. as-broadcast script) must be structured into a sequence of dialogue lines each including time codes, speaker name and transcript. Current speech recognition technology alleviates the transcription step. However, state-of-the-art speaker diarization models still fall short on TV shows for two main reasons: (i) their inability to track a large number of speakers, (ii) their low accuracy in detecting frequent speaker changes. To mitigate this problem, we present a novel approach to leverage production scripts used during the shooting process, to extract pseudo-labeled data for the speaker diarization task. We propose a novel semi-supervised approach and demonstrate improvements of 51.7% relative to two unsupervised baseline models on our metrics on a 66 show test set.