# Factorised Speaker-environment Adaptive Training of Conformer Speech Recognition Systems

arXiv: https://arxiv.org/abs/2306.14608 · HTML: https://mlsanity.com/paper/2306.14608/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 56 references that are in the map.
- First posted on arXiv 26 Jun 2023.

## Builds on (most cited first, 21 of 21)

- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [WHAM!: Extending Speech Separation to Noisy Environments](https://mlsanity.com/paper/1907.01160/) (2019)
- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [Recent Developments on ESPnet Toolkit Boosted by Conformer](https://mlsanity.com/paper/2010.13956/) (2020)
- [Multi-task self-supervised learning for Robust Speech Recognition](https://mlsanity.com/paper/2001.09239/) (2020)
- [End-to-End Multi-speaker Speech Recognition with Transformer](https://mlsanity.com/paper/2002.03921/) (2020)
- [Learning Hidden Unit Contributions for Unsupervised Acoustic Model Adaptation](https://mlsanity.com/paper/1601.02828/) (2016)
- [On the limit of English conversational speech recognition](https://mlsanity.com/paper/2105.00982/) (2021)
- [Speaker adaptation for Wav2vec2 based dysarthric ASR](https://mlsanity.com/paper/2204.00770/) (2022)
- [Transformer Language Models with LSTM-based Cross-utterance Information Representation](https://mlsanity.com/paper/2102.06474/) (2021)
- [Conformer-based Hybrid ASR System for Switchboard Dataset](https://mlsanity.com/paper/2111.03442/) (2021)
- [Unsupervised Speaker Adaptation using Attention-based Speaker Memory for End-to-End ASR](https://mlsanity.com/paper/2002.06165/) (2020)
- [End-to-End Dereverberation, Beamforming, and Speech Recognition with Improved Numerical Stability and Advanced Frontend](https://mlsanity.com/paper/2102.11525/) (2021)
- [Cumulative Adaptation for BLSTM Acoustic Models](https://mlsanity.com/paper/1906.06207/) (2019)
- [Audio-visual Multi-channel Integration and Recognition of Overlapped Speech](https://mlsanity.com/paper/2011.07755/) (2020)
- [Bayesian Learning for Deep Neural Network Adaptation](https://mlsanity.com/paper/2012.07460/) (2020)
- [Improving the Training Recipe for a Robust Conformer-based Hybrid Model](https://mlsanity.com/paper/2206.12955/) (2022)
- [End-to-End Speech Recognition With Joint Dereverberation Of Sub-Band Autoregressive Envelopes](https://mlsanity.com/paper/2108.03975/) (2021)
- [Domain Adaptation of low-resource Target-Domain models using well-trained ASR Conformer Models](https://mlsanity.com/paper/2202.09167/) (2022)
- [Confidence Score Based Speaker Adaptation of Conformer Speech Recognition Systems](https://mlsanity.com/paper/2302.07521/) (2023)

## Abstract (arXiv)

Rich sources of variability in natural speech present significant challenges to current data intensive speech recognition technologies. To model both speaker and environment level diversity, this paper proposes a novel Bayesian factorised speaker-environment adaptive training and test time adaptation approach for Conformer ASR models. Speaker and environment level characteristics are separately modeled using compact hidden output transforms, which are then linearly or hierarchically combined to represent any speaker-environment combination. Bayesian learning is further utilized to model the adaptation parameter uncertainty. Experiments on the 300-hr WHAM noise corrupted Switchboard data suggest that factorised adaptation consistently outperforms the baseline and speaker label only adapted Conformers by up to 3.1% absolute (10.4% relative) word error rate reductions. Further analysis shows the proposed method offers potential for rapid adaption to unseen speaker-environment conditions.