# Strategies for improving low resource speech to text translation relying on pre-trained ASR models

arXiv: https://arxiv.org/abs/2306.00208 · HTML: https://mlsanity.com/paper/2306.00208/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 28 references that are in the map.
- First posted on arXiv 31 May 2023.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)
- [How2: A Large-scale Dataset for Multimodal Language Understanding](https://mlsanity.com/paper/1811.00347/) (2018)
- [ESPnet-ST: All-in-One Speech Translation Toolkit](https://mlsanity.com/paper/2004.10234/) (2020)
- [Pre-training on high-resource speech recognition improves low-resource speech-to-text translation](https://mlsanity.com/paper/1809.01431/) (2018)
- [Cascade versus Direct Speech Translation: Do the Differences Still Make a Difference?](https://mlsanity.com/paper/2106.01045/) (2021)
- [Analyzing ASR pretraining for low-resource speech-to-text translation](https://mlsanity.com/paper/1910.10762/) (2019)
- [Revisiting End-to-End Speech-to-Text Translation From Scratch](https://mlsanity.com/paper/2206.04571/) (2022)
- [CTC Alignments Improve Autoregressive Translation](https://mlsanity.com/paper/2210.05200/) (2022)
- [Searchable Hidden Intermediates for End-to-End Models of Decomposable Sequence Tasks](https://mlsanity.com/paper/2105.00573/) (2021)
- [Investigating the Reordering Capability in CTC-based Non-Autoregressive End-to-End Speech Translation](https://mlsanity.com/paper/2105.04840/) (2021)
- [Speech Resources in the Tamasheq Language](https://mlsanity.com/paper/2201.05051/) (2022)
- [Jointly Trained Transformers models for Spoken Language Translation](https://mlsanity.com/paper/2004.12111/) (2020)
- [Does Joint Training Really Help Cascaded Speech Translation?](https://mlsanity.com/paper/2210.13700/) (2022)
- [Cross-lingual topic prediction for speech using translations](https://mlsanity.com/paper/1908.11425/) (2019)

## Built on by (most cited first, 3 of 3)

- [IIITH-BUT system for IWSLT 2025 low-resource Bhojpuri to Hindi speech translation](https://mlsanity.com/paper/2506.04714/) (2025)
- [End-to-End Speech Translation for Low-Resource Languages Using Weakly Labeled Data](https://mlsanity.com/paper/2506.16251/) (2025)
- [Speech-to-Speech Translation Pipelines for Conversations in Low-Resource Languages](https://mlsanity.com/paper/2506.01406/) (2025)

## Abstract (arXiv)

This paper presents techniques and findings for improving the performance of low-resource speech to text translation (ST). We conducted experiments on both simulated and real-low resource setups, on language pairs English - Portuguese, and Tamasheq - French respectively. Using the encoder-decoder framework for ST, our results show that a multilingual automatic speech recognition system acts as a good initialization under low-resource scenarios. Furthermore, using the CTC as an additional objective for translation during training and decoding helps to reorder the internal representations and improves the final translation. Through our experiments, we try to identify various factors (initializations, objectives, and hyper-parameters) that contribute the most for improvements in low-resource setups. With only 300 hours of pre-training data, our model achieved 7.3 BLEU score on Tamasheq - French data, outperforming prior published works from IWSLT 2022 by 1.6 points.