# Analyzing And Improving Neural Speaker Embeddings for ASR

arXiv: https://arxiv.org/abs/2301.04571 · HTML: https://mlsanity.com/paper/2301.04571/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 38 references that are in the map.
- First posted on arXiv 11 Jan 2023.

## Builds on (most cited first, 15 of 15)

- [Attentive Statistics Pooling for Deep Speaker Embedding](https://mlsanity.com/paper/1803.10963/) (2018)
- [RWTH ASR Systems for LibriSpeech: Hybrid vs Attention -- w/o Data Augmentation](https://mlsanity.com/paper/1905.03072/) (2019)
- [MFA-Conformer: Multi-scale Feature Aggregation Conformer for Automatic Speaker Verification](https://mlsanity.com/paper/2203.15249/) (2022)
- [Learning Hidden Unit Contributions for Unsupervised Acoustic Model Adaptation](https://mlsanity.com/paper/1601.02828/) (2016)
- [RawNet: Advanced end-to-end deep neural network using raw waveforms for text-independent speaker verification](https://mlsanity.com/paper/1904.08104/) (2019)
- [RETURNN as a Generic Flexible Neural Toolkit with Application to Translation and Speech Recognition](https://mlsanity.com/paper/1805.05225/) (2018)
- [On the limit of English conversational speech recognition](https://mlsanity.com/paper/2105.00982/) (2021)
- [Attention-Based Models for Text-Dependent Speaker Verification](https://mlsanity.com/paper/1710.10470/) (2017)
- [Efficient Training of Neural Transducer for Speech Recognition](https://mlsanity.com/paper/2204.10586/) (2022)
- [Cumulative Adaptation for BLSTM Acoustic Models](https://mlsanity.com/paper/1906.06207/) (2019)
- [LSTM Language Models for LVCSR in First-Pass Decoding and Lattice-Rescoring](https://mlsanity.com/paper/1907.01030/) (2019)
- [Improving the Training Recipe for a Robust Conformer-based Hybrid Model](https://mlsanity.com/paper/2206.12955/) (2022)
- [Deep Speaker Embedding Learning with Multi-Level Pooling for Text-Independent Speaker Verification](https://mlsanity.com/paper/1902.07821/) (2019)
- [Bayesian Learning of LF-MMI Trained Time Delay Neural Networks for Speech Recognition](https://mlsanity.com/paper/2012.04494/) (2020)
- [Embeddings for DNN speaker adaptive training](https://mlsanity.com/paper/1909.13537/) (2019)

## Abstract (arXiv)

Neural speaker embeddings encode the speaker's speech characteristics through a DNN model and are prevalent for speaker verification tasks. However, few studies have investigated the usage of neural speaker embeddings for an ASR system. In this work, we present our efforts w.r.t integrating neural speaker embeddings into a conformer based hybrid HMM ASR system. For ASR, our improved embedding extraction pipeline in combination with the Weighted-Simple-Add integration method results in x-vector and c-vector reaching on par performance with i-vectors. We further compare and analyze different speaker embeddings. We present our acoustic model improvements obtained by switching from newbob learning rate schedule to one cycle learning schedule resulting in a ~3% relative WER reduction on Switchboard, additionally reducing the overall training time by 17%. By further adding neural speaker embeddings, we gain additional ~3% relative WER improvement on Hub5'00. Our best Conformer-based hybrid ASR system with speaker embeddings achieves 9.0% WER on Hub5'00 and Hub5'01 with training on SWB 300h.