# Streaming Target-Speaker ASR with Neural Transducer

arXiv: https://arxiv.org/abs/2209.04175 · HTML: https://mlsanity.com/paper/2209.04175/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Sep 2022.

## Built on by (most cited first, 9 of 9)

- [Zero-shot text-to-speech synthesis conditioned using self-supervised speech representation model](https://mlsanity.com/paper/2304.11976/) (2023)
- [Conformer-based Target-Speaker Automatic Speech Recognition for Single-Channel Audio](https://mlsanity.com/paper/2308.05218/) (2023)
- [End-to-End Joint Target and Non-Target Speakers ASR](https://mlsanity.com/paper/2306.02273/) (2023)
- [Alignment-Free Training for Transducer-based Multi-Talker ASR](https://mlsanity.com/paper/2409.20301/) (2024)
- [TS-SUPERB: A Target Speech Processing Benchmark for Speech Self-Supervised Learning Models](https://mlsanity.com/paper/2505.06660/) (2025)
- [Knowledge Distillation for Neural Transducer-based Target-Speaker ASR: Exploiting Parallel Mixture/Single-Talker Speech Data](https://mlsanity.com/paper/2305.15971/) (2023)
- [A Glance is Enough: Extract Target Sentence By Looking at A keyword](https://mlsanity.com/paper/2310.05352/) (2023)
- [Hypothesis Clustering and Merging: Novel MultiTalker Speech Recognition with Speaker Tokens](https://mlsanity.com/paper/2409.15732/) (2024)
- [CALM: Joint Contextual Acoustic-Linguistic Modeling for Personalization of Multi-Speaker ASR](https://mlsanity.com/paper/2601.22792/) (2026)

## Abstract (arXiv)

Although recent advances in deep learning technology have boosted automatic speech recognition (ASR) performance in the single-talker case, it remains difficult to recognize multi-talker speech in which many voices overlap. One conventional approach to tackle this problem is to use a cascade of a speech separation or target speech extraction front-end with an ASR back-end. However, the extra computation costs of the front-end module are a critical barrier to quick response, especially for streaming ASR. In this paper, we propose a target-speaker ASR (TS-ASR) system that implicitly integrates the target speech extraction functionality within a streaming end-to-end (E2E) ASR system, i.e. recurrent neural network-transducer (RNNT). Our system uses a similar idea as adopted for target speech extraction, but implements it directly at the level of the encoder of RNNT. This allows TS-ASR to be realized without placing extra computation costs on the front-end. Note that this study presents two major differences between prior studies on E2E TS-ASR; we investigate streaming models and base our study on Conformer models, whereas prior studies used RNN-based systems and considered only offline processing. We confirm in experiments that our TS-ASR achieves comparable recognition performance with conventional cascade systems in the offline setting, while reducing computation costs and realizing streaming TS-ASR.