# Augmenting Transformer-Transducer Based Speaker Change Detection With Token-Level Training Loss

arXiv: https://arxiv.org/abs/2211.06482 · HTML: https://mlsanity.com/paper/2211.06482/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 33 references that are in the map.
- First posted on arXiv 11 Nov 2022.

## Builds on (most cited first, 7 of 7)

- [In-Datacenter Performance Analysis of a Tensor Processing Unit](https://mlsanity.com/paper/1704.04760/) (2017)
- [Generalized End-to-End Loss for Speaker Verification](https://mlsanity.com/paper/1710.10467/) (2017)
- [Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss](https://mlsanity.com/paper/2002.02562/) (2020)
- [Joint Speech Recognition and Speaker Diarization via Sequence Transduction](https://mlsanity.com/paper/1907.05337/) (2019)
- [Turn-to-Diarize: Online Speaker Diarization Constrained by Transformer Transducer Speaker Turn Detection](https://mlsanity.com/paper/2109.11641/) (2021)
- [Optimizing expected word error rate via sampling for speech recognition](https://mlsanity.com/paper/1706.02776/) (2017)
- [Highly Efficient Real-Time Streaming and Fully On-Device Speaker Diarization with Multi-Stage Clustering](https://mlsanity.com/paper/2210.13690/) (2022)

## Built on by (most cited first, 8 of 8)

- [Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamics](https://mlsanity.com/paper/2503.01174/) (2025)
- [DiarizationLM: Speaker Diarization Post-Processing with Large Language Models](https://mlsanity.com/paper/2401.03506/) (2024)
- [LLM-based speaker diarization correction: A generalizable approach](https://mlsanity.com/paper/2406.04927/) (2024)
- [USM-SCD: Multilingual Speaker Change Detection Based on Large Pretrained Foundation Models](https://mlsanity.com/paper/2309.08023/) (2023)
- [Improved Long-Form Speech Recognition by Jointly Modeling the Primary and Non-primary Speakers](https://mlsanity.com/paper/2312.11123/) (2023)
- [Encoder-decoder multimodal speaker change detection](https://mlsanity.com/paper/2306.00680/) (2023)
- [Towards Word-Level End-to-End Neural Speaker Diarization with Auxiliary Network](https://mlsanity.com/paper/2309.08489/) (2023)
- [Leveraging Speaker Embeddings in End-to-End Neural Diarization for Two-Speaker Scenarios](https://mlsanity.com/paper/2407.01317/) (2024)

## Abstract (arXiv)

In this work we propose a novel token-based training strategy that improves Transformer-Transducer (T-T) based speaker change detection (SCD) performance. The conventional T-T based SCD model loss optimizes all output tokens equally. Due to the sparsity of the speaker changes in the training data, the conventional T-T based SCD model loss leads to sub-optimal detection accuracy. To mitigate this issue, we use a customized edit-distance algorithm to estimate the token-level SCD false accept (FA) and false reject (FR) rates during training and optimize model parameters to minimize a weighted combination of the FA and FR, focusing the model on accurately predicting speaker changes. We also propose a set of evaluation metrics that align better with commercial use cases. Experiments on a group of challenging real-world datasets show that the proposed training method can significantly improve the overall performance of the SCD model with the same number of parameters.