# Unsupervised Speaker Adaptation using Attention-based Speaker Memory for End-to-End ASR

arXiv: https://arxiv.org/abs/2002.06165 · HTML: https://mlsanity.com/paper/2002.06165/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Feb 2020.

## Built on by (most cited first, 12 of 12)

- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [End-to-End Speech Recognition: A Survey](https://mlsanity.com/paper/2303.03329/) (2023)
- [Advanced Long-context End-to-end Speech Recognition Using Context-expanded Transformers](https://mlsanity.com/paper/2104.09426/) (2021)
- [Text is All You Need: Personalizing ASR Models using Controllable Speech Synthesis](https://mlsanity.com/paper/2303.14885/) (2023)
- [Is Lip Region-of-Interest Sufficient for Lipreading?](https://mlsanity.com/paper/2205.14295/) (2022)
- [Multiple-hypothesis CTC-based semi-supervised adaptation of end-to-end speech recognition](https://mlsanity.com/paper/2103.15515/) (2021)
- [Target Speaker Lipreading by Audio-Visual Self-Distillation Pretraining and Speaker Adaptation](https://mlsanity.com/paper/2502.05758/) (2025)
- [Enhancing Quantised End-to-End ASR Models via Personalisation](https://mlsanity.com/paper/2309.09136/) (2023)
- [MobileASR: A resource-aware on-device learning framework for user voice personalization applications on mobile phones](https://mlsanity.com/paper/2306.09384/) (2023)
- [Factorised Speaker-environment Adaptive Training of Conformer Speech Recognition Systems](https://mlsanity.com/paper/2306.14608/) (2023)
- [Memory-augmented conformer for improved end-to-end long-form ASR](https://mlsanity.com/paper/2309.13029/) (2023)
- [A Probe Shift Is Not a Fairness Fix: The Limits of Representation Steering in Speech Models](https://mlsanity.com/paper/2609.18533/) (2026)

## Abstract (arXiv)

We propose an unsupervised speaker adaptation method inspired by the neural Turing machine for end-to-end (E2E) automatic speech recognition (ASR). The proposed model contains a memory block that holds speaker i-vectors extracted from the training data and reads relevant i-vectors from the memory through an attention mechanism. The resulting memory vector (M-vector) is concatenated to the acoustic features or to the hidden layer activations of an E2E neural network model. The E2E ASR system is based on the joint connectionist temporal classification and attention-based encoder-decoder architecture. M-vector and i-vector results are compared for inserting them at different layers of the encoder neural network using the WSJ and TED-LIUM2 ASR benchmarks. We show that M-vectors, which do not require an auxiliary speaker embedding extraction system at test time, achieve similar word error rates (WERs) compared to i-vectors for single speaker utterances and significantly lower WERs for utterances in which there are speaker changes.