# Multi-pass Training and Cross-information Fusion for Low-resource End-to-end Accented Speech Recognition

arXiv: https://arxiv.org/abs/2306.11309 · HTML: https://mlsanity.com/paper/2306.11309/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 33 references that are in the map.
- First posted on arXiv 20 Jun 2023.

## Builds on (most cited first, 24 of 24)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baseline](https://mlsanity.com/paper/1709.05522/) (2017)
- [State-of-the-art Speech Recognition With Sequence-to-Sequence Models](https://mlsanity.com/paper/1712.01769/) (2017)
- [Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition](https://mlsanity.com/paper/2010.10504/) (2020)
- [End-to-End Attention-based Large Vocabulary Speech Recognition](https://mlsanity.com/paper/1508.04395/) (2015)
- [Recent Developments on ESPnet Toolkit Boosted by Conformer](https://mlsanity.com/paper/2010.13956/) (2020)
- [A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillation](https://mlsanity.com/paper/1810.13243/) (2018)
- [Personalizing ASR for Dysarthric and Accented Speech with Limited Data](https://mlsanity.com/paper/1907.13511/) (2019)
- [Multi-Dialect Speech Recognition With A Single Sequence-To-Sequence Model](https://mlsanity.com/paper/1712.01541/) (2017)
- [Conformer-based Hybrid ASR System for Switchboard Dataset](https://mlsanity.com/paper/2111.03442/) (2021)
- [Improving Accent Identification and Accented Speech Recognition Under a Framework of Self-supervised Learning](https://mlsanity.com/paper/2109.07349/) (2021)
- [E2E-based Multi-task Learning Approach to Joint Speech and Accent Recognition](https://mlsanity.com/paper/2106.08211/) (2021)
- [Layer-wise Fast Adaptation for End-to-End Multi-Accent Speech Recognition](https://mlsanity.com/paper/2204.09883/) (2022)
- [Conformer with dual-mode chunked attention for joint online and offline ASR](https://mlsanity.com/paper/2206.11157/) (2022)
- [Confidence Score Based Conformer Speaker Adaptation for Speech Recognition](https://mlsanity.com/paper/2206.12045/) (2022)

## Built on by (most cited first, 3 of 3)

- [Improving Accented Speech Recognition using Data Augmentation based on Unsupervised Text-to-Speech Synthesis](https://mlsanity.com/paper/2407.04047/) (2024)
- [Qifusion-Net: Layer-adapted Stream/Non-stream Model for End-to-End Multi-Accent Speech Recognition](https://mlsanity.com/paper/2407.03026/) (2024)
- [Phonetic Modeling of Dialectal Variation in Vietnamese Speech](https://mlsanity.com/paper/2605.24451/) (2026)

## Abstract (arXiv)

Low-resource accented speech recognition is one of the important challenges faced by current ASR technology in practical applications. In this study, we propose a Conformer-based architecture, called Aformer, to leverage both the acoustic information from large non-accented and limited accented training data. Specifically, a general encoder and an accent encoder are designed in the Aformer to extract complementary acoustic information. Moreover, we propose to train the Aformer in a multi-pass manner, and investigate three cross-information fusion methods to effectively combine the information from both general and accent encoders. All experiments are conducted on both the accented English and Mandarin ASR tasks. Results show that our proposed methods outperform the strong Conformer baseline by relative 10.2% to 24.5% word/character error rate reduction on six in-domain and out-of-domain accented test sets.