# AlloST: Low-resource Speech Translation without Source Transcription

arXiv: https://arxiv.org/abs/2105.00171 · HTML: https://mlsanity.com/paper/2105.00171/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 31 references that are in the map.
- First posted on arXiv 1 May 2021.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [BPE-Dropout: Simple and Effective Subword Regularization](https://mlsanity.com/paper/1910.13267/) (2019)
- [Sequence-to-Sequence Models Can Directly Translate Foreign Speech](https://mlsanity.com/paper/1703.08581/) (2017)
- [Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translation](https://mlsanity.com/paper/1612.01744/) (2016)
- [ESPnet-ST: All-in-One Speech Translation Toolkit](https://mlsanity.com/paper/2004.10234/) (2020)
- [Universal Phone Recognition with a Multilingual Allophone System](https://mlsanity.com/paper/2002.11800/) (2020)
- [End-to-End Speech Translation with Knowledge Distillation](https://mlsanity.com/paper/1904.08075/) (2019)
- [Curriculum Pre-training for End-to-End Speech Translation](https://mlsanity.com/paper/2004.10093/) (2020)
- [Tied Multitask Learning for Neural Speech Translation](https://mlsanity.com/paper/1802.06655/) (2018)
- [Dual-decoder Transformer for Joint Automatic Speech Recognition and Multilingual Speech Translation](https://mlsanity.com/paper/2011.00747/) (2020)
- [Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech Translation](https://mlsanity.com/paper/1909.07575/) (2019)
- [Attention-Passing Models for Robust and Data-Efficient End-to-End Speech Translation](https://mlsanity.com/paper/1904.07209/) (2019)
- [Input Combination Strategies for Multi-Source Transformer Decoder](https://mlsanity.com/paper/1811.04716/) (2018)
- [Exploring Phoneme-Level Speech Representations for End-to-End Speech Translation](https://mlsanity.com/paper/1906.01199/) (2019)
- [Worse WER, but Better BLEU? Leveraging Word Embedding as Intermediate in Multitask End-to-End Speech Translation](https://mlsanity.com/paper/2005.10678/) (2020)
- [Phone Features Improve Speech Translation](https://mlsanity.com/paper/2005.13681/) (2020)
- [Unwritten Languages Demand Attention Too! Word Discovery with Encoder-Decoder Models](https://mlsanity.com/paper/1709.05631/) (2017)

## Built on by (most cited first, 2 of 2)

- [Simple and Effective Unsupervised Speech Translation](https://mlsanity.com/paper/2210.10191/) (2022)
- [How to Evaluate Speech Translation with Source-Aware Neural MT Metrics](https://mlsanity.com/paper/2511.03295/) (2025)

## Abstract (arXiv)

The end-to-end architecture has made promising progress in speech translation (ST). However, the ST task is still challenging under low-resource conditions. Most ST models have shown unsatisfactory results, especially in the absence of word information from the source speech utterance. In this study, we survey methods to improve ST performance without using source transcription, and propose a learning framework that utilizes a language-independent universal phone recognizer. The framework is based on an attention-based sequence-to-sequence model, where the encoder generates the phonetic embeddings and phone-aware acoustic representations, and the decoder controls the fusion of the two embedding streams to produce the target token sequence. In addition to investigating different fusion strategies, we explore the specific usage of byte pair encoding (BPE), which compresses a phone sequence into a syllable-like segmented sequence. Due to the conversion of symbols, a segmented sequence represents not only pronunciation but also language-dependent information lacking in phones. Experiments conducted on the Fisher Spanish-English and Taigi-Mandarin drama corpora show that our method outperforms the conformer-based baseline, and the performance is close to that of the existing best method using source transcription.