# E2E-based Multi-task Learning Approach to Joint Speech and Accent Recognition

arXiv: https://arxiv.org/abs/2106.08211 · HTML: https://mlsanity.com/paper/2106.08211/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jun 2021.

## Built on by (most cited first, 12 of 12)

- [Accented Speech Recognition With Accent-specific Codebooks](https://mlsanity.com/paper/2310.15970/) (2023)
- [PM-MMUT: Boosted Phone-Mask Data Augmentation using Multi-Modeling Unit Training for Phonetic-Reduction-Robust E2E Speech Recognition](https://mlsanity.com/paper/2112.06721/) (2021)
- [Multi-pass Training and Cross-information Fusion for Low-resource End-to-end Accented Speech Recognition](https://mlsanity.com/paper/2306.11309/) (2023)
- [Towards spoken dialect identification of Irish](https://mlsanity.com/paper/2307.07436/) (2023)
- [AccentFold: A Journey through African Accents for Zero-Shot ASR Adaptation to Target Accents](https://mlsanity.com/paper/2402.01152/) (2024)
- [Transducer-based language embedding for spoken language identification](https://mlsanity.com/paper/2204.03888/) (2022)
- [Towards Word-Level End-to-End Neural Speaker Diarization with Auxiliary Network](https://mlsanity.com/paper/2309.08489/) (2023)
- [Generative linguistic representation for spoken language identification](https://mlsanity.com/paper/2312.10964/) (2023)
- [Improving Self-supervised Pre-training using Accent-Specific Codebooks](https://mlsanity.com/paper/2407.03734/) (2024)
- [Mixture-of-Experts with Intermediate CTC Supervision for Accented Speech Recognition](https://mlsanity.com/paper/2602.01967/) (2026)
- [Audio-AdapterFusion: A Task-ID-free Approach for Efficient and Non-Destructive Multi-task Speech Recognition](https://mlsanity.com/paper/2310.13015/) (2023)
- [Clustering and Mining Accented Speech for Inclusive and Fair Speech Recognition](https://mlsanity.com/paper/2408.02582/) (2024)

## Abstract (arXiv)

In this paper, we propose a single multi-task learning framework to perform End-to-End (E2E) speech recognition (ASR) and accent recognition (AR) simultaneously. The proposed framework is not only more compact but can also yield comparable or even better results than standalone systems. Specifically, we found that the overall performance is predominantly determined by the ASR task, and the E2E-based ASR pretraining is essential to achieve improved performance, particularly for the AR task. Additionally, we conduct several analyses of the proposed method. First, though the objective loss for the AR task is much smaller compared with its counterpart of ASR task, a smaller weighting factor with the AR task in the joint objective function is necessary to yield better results for each task. Second, we found that sharing only a few layers of the encoder yields better AR results than sharing the overall encoder. Experimentally, the proposed method produces WER results close to the best standalone E2E ASR ones, while it achieves 7.7% and 4.2% relative improvement over standalone and single-task-based joint recognition methods on test set for accent recognition respectively.