# Multi-task Learning for Speaker Verification and Voice Trigger Detection

arXiv: https://arxiv.org/abs/2001.10816 · HTML: https://mlsanity.com/paper/2001.10816/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jan 2020.

## Built on by (most cited first, 9 of 9)

- [Deep Spoken Keyword Spotting: An Overview](https://mlsanity.com/paper/2111.10592/) (2021)
- [Improving the Training Recipe for a Robust Conformer-based Hybrid Model](https://mlsanity.com/paper/2206.12955/) (2022)
- [UniverSLU: Universal Spoken Language Understanding for Diverse Tasks with Natural Language Instructions](https://mlsanity.com/paper/2310.02973/) (2023)
- [Multi-task learning for virtual flow metering](https://mlsanity.com/paper/2103.08713/) (2021)
- [Multimodal Data and Resource Efficient Device-Directed Speech Detection with Large Foundation Models](https://mlsanity.com/paper/2312.03632/) (2023)
- [A Multimodal Approach to Device-Directed Speech Detection with Large Language Models](https://mlsanity.com/paper/2403.14438/) (2024)
- [Improving Voice Trigger Detection with Metric Learning](https://mlsanity.com/paper/2204.02455/) (2022)
- [End-To-End Audiovisual Feature Fusion for Active Speaker Detection](https://mlsanity.com/paper/2207.13434/) (2022)
- [A Multi-tasking Model of Speaker-Keyword Classification for Keeping Human in the Loop of Drone-assisted Inspection](https://mlsanity.com/paper/2207.04027/) (2022)

## Abstract (arXiv)

Automatic speech transcription and speaker recognition are usually treated as separate tasks even though they are interdependent. In this study, we investigate training a single network to perform both tasks jointly. We train the network in a supervised multi-task learning setup, where the speech transcription branch of the network is trained to minimise a phonetic connectionist temporal classification (CTC) loss while the speaker recognition branch of the network is trained to label the input sequence with the correct label for the speaker. We present a large-scale empirical study where the model is trained using several thousand hours of labelled training data for each task. We evaluate the speech transcription branch of the network on a voice trigger detection task while the speaker recognition branch is evaluated on a speaker verification task. Results demonstrate that the network is able to encode both phonetic and speaker information in its learnt representations while yielding accuracies at least as good as the baseline models for each task, with the same number of parameters as the independent models.