# Joint Modeling of Accents and Acoustics for Multi-Accent Speech Recognition

arXiv: https://arxiv.org/abs/1802.02656 · HTML: https://mlsanity.com/paper/1802.02656/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Feb 2018.

## Built on by (most cited first, 14 of 14)

- [Can Whisper perform speech-based in-context learning?](https://mlsanity.com/paper/2309.07081/) (2023)
- [Streaming End-to-End Multilingual Speech Recognition with Joint Language Identification](https://mlsanity.com/paper/2209.06058/) (2022)
- [Algorithmic Bias and Data Bias: Understanding the Relation between Distributionally Robust Optimization and Data Curation](https://mlsanity.com/paper/2106.09467/) (2021)
- [Accented Speech Recognition: A Survey](https://mlsanity.com/paper/2104.10747/) (2021)
- [Supervised Contrastive Learning for Accented Speech Recognition](https://mlsanity.com/paper/2107.00921/) (2021)
- [Enhancing and Adversarial: Improve ASR with Speaker Labels](https://mlsanity.com/paper/2211.06369/) (2022)
- [Towards dialect-inclusive recognition in a low-resource language: are balanced corpora the answer?](https://mlsanity.com/paper/2307.07295/) (2023)
- [A Highly Adaptive Acoustic Model for Accurate Multi-Dialect Speech Recognition](https://mlsanity.com/paper/2205.03027/) (2022)
- [INTapt: Information-Theoretic Adversarial Prompt Tuning for Enhanced Non-Native Speech Recognition](https://mlsanity.com/paper/2305.16371/) (2023)
- [Towards spoken dialect identification of Irish](https://mlsanity.com/paper/2307.07436/) (2023)
- [Pushing the performances of ASR models on English and Spanish accents](https://mlsanity.com/paper/2212.12048/) (2022)
- [A meta learning scheme for fast accent domain expansion in Mandarin speech recognition](https://mlsanity.com/paper/2307.12262/) (2023)
- [Clustering and Mining Accented Speech for Inclusive and Fair Speech Recognition](https://mlsanity.com/paper/2408.02582/) (2024)
- [Proficiency-Aware Adaptation and Data Augmentation for Robust L2 ASR](https://mlsanity.com/paper/2510.10738/) (2025)

## Abstract (arXiv)

The performance of automatic speech recognition systems degrades with increasing mismatch between the training and testing scenarios. Differences in speaker accents are a significant source of such mismatch. The traditional approach to deal with multiple accents involves pooling data from several accents during training and building a single model in multi-task fashion, where tasks correspond to individual accents. In this paper, we explore an alternate model where we jointly learn an accent classifier and a multi-task acoustic model. Experiments on the American English Wall Street Journal and British English Cambridge corpora demonstrate that our joint model outperforms the strong multi-task acoustic model baseline. We obtain a 5.94% relative improvement in word error rate on British English, and 9.47% relative improvement on American English. This illustrates that jointly modeling with accent information improves acoustic model performance.