# Accented Speech Recognition With Accent-specific Codebooks

arXiv: https://arxiv.org/abs/2310.15970 · HTML: https://mlsanity.com/paper/2310.15970/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 57 references that are in the map.
- First posted on arXiv 24 Oct 2023.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learning](https://mlsanity.com/paper/1609.06773/) (2016)
- [Self-supervised Learning with Random-projection Quantizer for Speech Recognition](https://mlsanity.com/paper/2202.01855/) (2022)
- [Domain Adversarial Training for Accented Speech Recognition](https://mlsanity.com/paper/1806.02786/) (2018)
- [Best-First Beam Search](https://mlsanity.com/paper/2007.03909/) (2020)
- [Multi-Dialect Speech Recognition With A Single Sequence-To-Sequence Model](https://mlsanity.com/paper/1712.01541/) (2017)
- [Learning Fast Adaptation on Cross-Accented Speech Recognition](https://mlsanity.com/paper/2003.01901/) (2020)
- [REDAT: Accent-Invariant Representation for End-to-End ASR by Domain Adversarial Training with Relabeling](https://mlsanity.com/paper/2012.07353/) (2020)
- [Best of Both Worlds: Robust Accented Speech Recognition with Adversarial Transfer Learning](https://mlsanity.com/paper/2103.05834/) (2021)
- [Accented Speech Recognition: Benchmarking, Pre-training, and Diverse Data](https://mlsanity.com/paper/2205.08014/) (2022)
- [Improving Accent Identification and Accented Speech Recognition Under a Framework of Self-supervised Learning](https://mlsanity.com/paper/2109.07349/) (2021)
- [Accent-Robust Automatic Speech Recognition Using Supervised and Unsupervised Wav2vec Embeddings](https://mlsanity.com/paper/2110.03520/) (2021)
- [E2E-based Multi-task Learning Approach to Joint Speech and Accent Recognition](https://mlsanity.com/paper/2106.08211/) (2021)
- [Supervised Contrastive Learning for Accented Speech Recognition](https://mlsanity.com/paper/2107.00921/) (2021)
- [Enhancing and Adversarial: Improve ASR with Speaker Labels](https://mlsanity.com/paper/2211.06369/) (2022)
- [A Highly Adaptive Acoustic Model for Accurate Multi-Dialect Speech Recognition](https://mlsanity.com/paper/2205.03027/) (2022)
- [Accented Speech Recognition Inspired by Human Perception](https://mlsanity.com/paper/2104.04627/) (2021)

## Built on by (most cited first, 6 of 6)

- [Improving Accented Speech Recognition using Data Augmentation based on Unsupervised Text-to-Speech Synthesis](https://mlsanity.com/paper/2407.04047/) (2024)
- [Adapting Automatic Speech Recognition for Accented Air Traffic Control Communications](https://mlsanity.com/paper/2502.20311/) (2025)
- [Effects of Speaker Count, Duration, and Accent Diversity on Zero-Shot Accent Robustness in Low-Resource ASR](https://mlsanity.com/paper/2506.04364/) (2025)
- [Improving Self-supervised Pre-training using Accent-Specific Codebooks](https://mlsanity.com/paper/2407.03734/) (2024)
- [Mixture-of-Experts with Intermediate CTC Supervision for Accented Speech Recognition](https://mlsanity.com/paper/2602.01967/) (2026)
- [Cross-Modal Robustness Transfer (CMRT): Training Robust Speech Translation Models Using Adversarial Text](https://mlsanity.com/paper/2602.11933/) (2026)

## Abstract (arXiv)

Speech accents pose a significant challenge to state-of-the-art automatic speech recognition (ASR) systems. Degradation in performance across underrepresented accents is a severe deterrent to the inclusive adoption of ASR. In this work, we propose a novel accent adaptation approach for end-to-end ASR systems using cross-attention with a trainable set of codebooks. These learnable codebooks capture accent-specific information and are integrated within the ASR encoder layers. The model is trained on accented English speech, while the test data also contained accents which were not seen during training. On the Mozilla Common Voice multi-accented dataset, we show that our proposed approach yields significant performance gains not only on the seen English accents (up to 37% relative improvement in word error rate) but also on the unseen accents (up to 5% relative improvement in WER). Further, we illustrate benefits for a zero-shot transfer setup on the L2Artic dataset. We also compare the performance with other approaches based on accent adversarial training.