# Pretraining by Backtranslation for End-to-end ASR in Low-Resource Settings

arXiv: https://arxiv.org/abs/1812.03919 · HTML: https://mlsanity.com/paper/1812.03919/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Dec 2018.

## Built on by (most cited first, 7 of 7)

- [A Survey of Data Augmentation Approaches for NLP](https://mlsanity.com/paper/2105.03075/) (2021)
- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [Improving Deliberation by Text-Only and Semi-Supervised Training](https://mlsanity.com/paper/2206.14716/) (2022)
- [On-the-Fly Aligned Data Augmentation for Sequence-to-Sequence ASR](https://mlsanity.com/paper/2104.01393/) (2021)
- [Decision Attentive Regularization to Improve Simultaneous Speech Translation Systems](https://mlsanity.com/paper/2110.15729/) (2021)
- [Two-Stage Augmentation and Adaptive CTC Fusion for Improved Robustness of Multi-Stream End-to-End ASR](https://mlsanity.com/paper/2102.03055/) (2021)
- [Improving Semi-supervised End-to-end Automatic Speech Recognition using CycleGAN and Inter-domain Losses](https://mlsanity.com/paper/2210.11642/) (2022)

## Abstract (arXiv)

We explore training attention-based encoder-decoder ASR in low-resource settings. These models perform poorly when trained on small amounts of transcribed speech, in part because they depend on having sufficient target-side text to train the attention and decoder networks. In this paper we address this shortcoming by pretraining our network parameters using only text-based data and transcribed speech from other languages. We analyze the relative contributions of both sources of data. Across 3 test languages, our text-based approach resulted in a 20% average relative improvement over a text-based augmentation technique without pretraining. Using transcribed speech from nearby languages gives a further 20-30% relative reduction in character error rate.