# End-to-End Target Speaker Speech Recognition Using Context-Aware Attention Mechanisms for Challenging Enrollment Scenario

arXiv: https://arxiv.org/abs/2501.15466 · HTML: https://mlsanity.com/paper/2501.15466/

## Key facts

- No citing papers in the map yet (data through 19 Oct 2026).
- Placed in Speaker recognition and diarization, one of 11 directions in Speech and audio language models.
- Builds on 13 of its 32 references that are in the map.
- First posted on arXiv 26 Jan 2025.

## Placed in

- Speech and audio language models / [Speaker recognition and diarization](https://mlsanity.com/speech/speaker-recognition-and-diarization/)

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [WHAM!: Extending Speech Separation to Noisy Environments](https://mlsanity.com/paper/1907.01160/) (2019)
- [VoiceFilter: Targeted Voice Separation by Speaker-Conditioned Spectrogram Masking](https://mlsanity.com/paper/1810.04826/) (2018)
- [Zipformer: A faster and better encoder for automatic speech recognition](https://mlsanity.com/paper/2310.11230/) (2023)
- [Listen to What You Want: Neural Network-based Universal Sound Selector](https://mlsanity.com/paper/2006.05712/) (2020)
- [End-to-End Neural Diarization: Reformulating Speaker Diarization as Simple Multi-label Classification](https://mlsanity.com/paper/2003.02966/) (2020)
- [Real-Time Target Sound Extraction](https://mlsanity.com/paper/2211.02250/) (2022)
- [Streaming Target-Speaker ASR with Neural Transducer](https://mlsanity.com/paper/2209.04175/) (2022)
- [Improving RNN Transducer With Target Speaker Extraction and Neural Uncertainty Estimation](https://mlsanity.com/paper/2011.13393/) (2020)
- [End-to-end Anchored Speech Recognition](https://mlsanity.com/paper/1902.02383/) (2019)
- [Anchored Speech Recognition with Neural Transducers](https://mlsanity.com/paper/2210.11588/) (2022)

## Abstract (arXiv)

This paper presents a novel streaming end-to-end target-speaker speech recognition that addresses two critical limitations in systems: the handling of noisy enrollment utterances and specific enrollment phrase requirements. This paper proposes a robust Target-Speaker Recurrent Neural Network Transducer (TS-RNNT) with dual attention mechanisms for contextual biasing and overlapping enrollment processing. The model incorporates a text decoder and attention mechanism specifically designed to extract relevant speaker characteristics from noisy, overlapping enrollment audio. Experimental results on a synthesized dataset demonstrate the model's resilience, maintaining a Word Error Rate (WER) of 16.44% even with overlapping enrollment at 5dB Signal-to-Interference Ratio (SIR), compared to conventional approaches that degrade to WERs above 75% under similar conditions. This significant performance improvement, coupled with the model's semi-text-dependent enrollment capabilities, represents a substantial advancement toward more practical and versatile voice-controlled devices.