# Improving RNN Transducer With Target Speaker Extraction and Neural Uncertainty Estimation

arXiv: https://arxiv.org/abs/2011.13393 · HTML: https://mlsanity.com/paper/2011.13393/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Nov 2020.

## Built on by (most cited first, 4 of 4)

- [uSee: Unified Speech Enhancement and Editing with Conditional Diffusion Models](https://mlsanity.com/paper/2310.00900/) (2023)
- [Multi-user VoiceFilter-Lite via Attentive Speaker Embedding](https://mlsanity.com/paper/2107.01201/) (2021)
- [Knowledge Distillation for Neural Transducer-based Target-Speaker ASR: Exploiting Parallel Mixture/Single-Talker Speech Data](https://mlsanity.com/paper/2305.15971/) (2023)
- [Closing the Gap between Single-User and Multi-User VoiceFilter-Lite](https://mlsanity.com/paper/2202.12169/) (2022)

## Abstract (arXiv)

Target-speaker speech recognition aims to recognize target-speaker speech from noisy environments with background noise and interfering speakers. This work presents a joint framework that combines time-domain target-speaker speech extraction and Recurrent Neural Network Transducer (RNN-T). To stabilize the joint-training, we propose a multi-stage training strategy that pre-trains and fine-tunes each module in the system before joint-training. Meanwhile, speaker identity and speech enhancement uncertainty measures are proposed to compensate for residual noise and artifacts from the target speech extraction module. Compared to a recognizer fine-tuned with a target speech extraction model, our experiments show that adding the neural uncertainty module significantly reduces 17% relative Character Error Rate (CER) on multi-speaker signals with background noise. The multi-condition experiments indicate that our method can achieve 9% relative performance gain in the noisy condition while maintaining the performance in the clean condition.