# Improving the Training Recipe for a Robust Conformer-based Hybrid Model

arXiv: https://arxiv.org/abs/2206.12955 · HTML: https://mlsanity.com/paper/2206.12955/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 42 references that are in the map.
- First posted on arXiv 26 Jun 2022.

## Builds on (most cited first, 23 of 23)

- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Attentive Statistics Pooling for Deep Speaker Embedding](https://mlsanity.com/paper/1803.10963/) (2018)
- [RWTH ASR Systems for LibriSpeech: Hybrid vs Attention -- w/o Data Augmentation](https://mlsanity.com/paper/1905.03072/) (2019)
- [Advancing RNN Transducer Technology for Speech Recognition](https://mlsanity.com/paper/2103.09935/) (2021)
- [Single headed attention based sequence-to-sequence model for state-of-the-art results on Switchboard](https://mlsanity.com/paper/2001.07263/) (2020)
- [RETURNN as a Generic Flexible Neural Toolkit with Application to Translation and Speech Recognition](https://mlsanity.com/paper/1805.05225/) (2018)
- [On the limit of English conversational speech recognition](https://mlsanity.com/paper/2105.00982/) (2021)
- [Self-Attentional Acoustic Models](https://mlsanity.com/paper/1803.09519/) (2018)
- [The RWTH ASR System for TED-LIUM Release 2: Improving Hybrid HMM with SpecAugment](https://mlsanity.com/paper/2004.00960/) (2020)
- [Conformer-based Hybrid ASR System for Switchboard Dataset](https://mlsanity.com/paper/2111.03442/) (2021)
- [Improving Accent Identification and Accented Speech Recognition Under a Framework of Self-supervised Learning](https://mlsanity.com/paper/2109.07349/) (2021)
- [Layer-wise Fast Adaptation for End-to-End Multi-Accent Speech Recognition](https://mlsanity.com/paper/2204.09883/) (2022)
- [Cumulative Adaptation for BLSTM Acoustic Models](https://mlsanity.com/paper/1906.06207/) (2019)
- [LSTM Language Models for LVCSR in First-Pass Decoding and Lattice-Rescoring](https://mlsanity.com/paper/1907.01030/) (2019)
- [Multi-task Learning for Speaker Verification and Voice Trigger Detection](https://mlsanity.com/paper/2001.10816/) (2020)
- [Multi-Accent Adaptation based on Gate Mechanism](https://mlsanity.com/paper/2011.02774/) (2020)
- [Bayesian Learning of LF-MMI Trained Time Delay Neural Networks for Speech Recognition](https://mlsanity.com/paper/2012.04494/) (2020)
- [A Conformer Based Acoustic Model for Robust Automatic Speech Recognition](https://mlsanity.com/paper/2203.00725/) (2022)
- [Embeddings for DNN speaker adaptive training](https://mlsanity.com/paper/1909.13537/) (2019)
- [A Highly Adaptive Acoustic Model for Accurate Multi-Dialect Speech Recognition](https://mlsanity.com/paper/2205.03027/) (2022)

## Built on by (most cited first, 9 of 9)

- [Enhancing and Adversarial: Improve ASR with Speaker Labels](https://mlsanity.com/paper/2211.06369/) (2022)
- [Towards Effective and Compact Contextual Representation for Conformer Transducer Speech Recognition Systems](https://mlsanity.com/paper/2306.13307/) (2023)
- [Uconv-Conformer: High Reduction of Input Sequence Length for End-to-End Speech Recognition](https://mlsanity.com/paper/2208.07657/) (2022)
- [Development of Hybrid ASR Systems for Low Resource Medical Domain Conversational Telephone Speech](https://mlsanity.com/paper/2210.13397/) (2022)
- [Unsupervised Pre-Training for Vietnamese Automatic Speech Recognition in the HYKIST Project](https://mlsanity.com/paper/2309.15869/) (2023)
- [Homogeneous Speaker Features for On-the-Fly Dysarthric and Elderly Speaker Adaptation](https://mlsanity.com/paper/2407.06310/) (2024)
- [End-to-End Training of a Neural HMM with Label and Transition Probabilities](https://mlsanity.com/paper/2310.02724/) (2023)
- [Analyzing And Improving Neural Speaker Embeddings for ASR](https://mlsanity.com/paper/2301.04571/) (2023)
- [Factorised Speaker-environment Adaptive Training of Conformer Speech Recognition Systems](https://mlsanity.com/paper/2306.14608/) (2023)

## Abstract (arXiv)

Speaker adaptation is important to build robust automatic speech recognition (ASR) systems. In this work, we investigate various methods for speaker adaptive training (SAT) based on feature-space approaches for a conformer-based acoustic model (AM) on the Switchboard 300h dataset. We propose a method, called Weighted-Simple-Add, which adds weighted speaker information vectors to the input of the multi-head self-attention module of the conformer AM. Using this method for SAT, we achieve 3.5% and 4.5% relative improvement in terms of WER on the CallHome part of Hub5'00 and Hub5'01 respectively. Moreover, we build on top of our previous work where we proposed a novel and competitive training recipe for a conformer-based hybrid AM. We extend and improve this recipe where we achieve 11% relative improvement in terms of word-error-rate (WER) on Switchboard 300h Hub5'00 dataset. We also make this recipe efficient by reducing the total number of parameters by 34% relative.