# Knowledge Distillation for Neural Transducer-based Target-Speaker ASR: Exploiting Parallel Mixture/Single-Talker Speech Data

arXiv: https://arxiv.org/abs/2305.15971 · HTML: https://mlsanity.com/paper/2305.15971/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation.
- Builds on 17 of its 39 references that are in the map.
- First posted on arXiv 25 May 2023.

## Placed in

- Knowledge distillation / [Speech and audio distillation](https://mlsanity.com/distill/speech-and-audio-distillation/)

## Builds on (most cited first, 17 of 17)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [Developing Real-time Streaming Transformer Transducer for Speech Recognition on Large-scale Dataset](https://mlsanity.com/paper/2010.11395/) (2020)
- [The fifth 'CHiME' Speech Separation and Recognition Challenge: Dataset, task and baselines](https://mlsanity.com/paper/1803.10609/) (2018)
- [Efficient Knowledge Distillation for RNN-Transducer Models](https://mlsanity.com/paper/2011.06110/) (2020)
- [VoiceFilter-Lite: Streaming Targeted Voice Separation for On-Device Speech Recognition](https://mlsanity.com/paper/2009.04323/) (2020)
- [Domain Adaptation via Teacher-Student Learning for End-to-End Speech Recognition](https://mlsanity.com/paper/2001.01798/) (2020)
- [Improving speaker discrimination of target speech extraction with time-domain SpeakerBeam](https://mlsanity.com/paper/2001.08378/) (2020)
- [Streaming Multi-speaker ASR with RNN-T](https://mlsanity.com/paper/2011.11671/) (2020)
- [Streaming Multi-talker Speech Recognition with Joint Speaker Identification](https://mlsanity.com/paper/2104.02109/) (2021)
- [Learning to Enhance or Not: Neural Network-Based Switching of Enhanced and Observed Signals for Overlapping Speech Recognition](https://mlsanity.com/paper/2201.03881/) (2022)
- [Streaming Target-Speaker ASR with Neural Transducer](https://mlsanity.com/paper/2209.04175/) (2022)
- [Should We Always Separate?: Switching Between Enhanced and Observed Signals for Overlapping Speech Recognition](https://mlsanity.com/paper/2106.00949/) (2021)
- [Improving RNN Transducer With Target Speaker Extraction and Neural Uncertainty Estimation](https://mlsanity.com/paper/2011.13393/) (2020)

## Built on by (most cited first, 3 of 3)

- [Empowering Whisper as a Joint Multi-Talker and Target-Talker Speech Recognition System](https://mlsanity.com/paper/2407.09817/) (2024)
- [Alignment-Free Training for Transducer-based Multi-Talker ASR](https://mlsanity.com/paper/2409.20301/) (2024)
- [Investigation of Speaker Representation for Target-Speaker Speech Processing](https://mlsanity.com/paper/2410.11243/) (2024)

## Abstract (arXiv)

Neural transducer (RNNT)-based target-speaker speech recognition (TS-RNNT) directly transcribes a target speaker's voice from a multi-talker mixture. It is a promising approach for streaming applications because it does not incur the extra computation costs of a target speech extraction frontend, which is a critical barrier to quick response. TS-RNNT is trained end-to-end given the input speech (i.e., mixtures and enrollment speech) and reference transcriptions. The training mixtures are generally simulated by mixing single-talker signals, but conventional TS-RNNT training does not utilize single-speaker signals. This paper proposes using knowledge distillation (KD) to exploit the parallel mixture/single-talker speech data. Our proposed KD scheme uses an RNNT system pretrained with the target single-talker speech input to generate pseudo labels for the TS-RNNT training. Experimental results show that TS-RNNT systems trained with the proposed KD scheme outperform a baseline TS-RNNT.