# Simple and Effective Unsupervised Speech Translation

arXiv: https://arxiv.org/abs/2210.10191 · HTML: https://mlsanity.com/paper/2210.10191/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 48 references that are in the map.
- First posted on arXiv 18 Oct 2022.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)
- [Word Translation Without Parallel Data](https://mlsanity.com/paper/1710.04087/) (2017)
- [VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://mlsanity.com/paper/2101.00390/) (2021)
- [Unsupervised Machine Translation Using Monolingual Corpora Only](https://mlsanity.com/paper/1711.00043/) (2017)
- [R-Drop: Regularized Dropout for Neural Networks](https://mlsanity.com/paper/2106.14448/) (2021)
- [fairseq S2T: Fast Speech-to-Text Modeling with fairseq](https://mlsanity.com/paper/2010.05171/) (2020)
- [Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training](https://mlsanity.com/paper/2104.01027/) (2021)
- [Unsupervised Neural Machine Translation](https://mlsanity.com/paper/1710.11041/) (2017)
- [Unsupervised Speech Recognition](https://mlsanity.com/paper/2105.11084/) (2021)
- [Sequence-to-Sequence Models Can Directly Translate Foreign Speech](https://mlsanity.com/paper/1703.08581/) (2017)
- [Direct speech-to-speech translation with a sequence-to-sequence model](https://mlsanity.com/paper/1904.06037/) (2019)
- [Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translation](https://mlsanity.com/paper/1612.01744/) (2016)
- [Translatotron 2: High-quality direct speech-to-speech translation with voice preservation](https://mlsanity.com/paper/2107.08661/) (2021)
- [CVSS Corpus and Massively Multilingual Speech-to-Speech Translation](https://mlsanity.com/paper/2201.03713/) (2022)
- [Pre-training on high-resource speech recognition improves low-resource speech-to-text translation](https://mlsanity.com/paper/1809.01431/) (2018)
- [Cascade versus Direct Speech Translation: Do the Differences Still Make a Difference?](https://mlsanity.com/paper/2106.01045/) (2021)
- [Towards End-to-end Unsupervised Speech Recognition](https://mlsanity.com/paper/2204.02492/) (2022)
- [Augmenting Librispeech with French Translations: A Multimodal Corpus for Direct Speech Translation Evaluation](https://mlsanity.com/paper/1802.03142/) (2018)
- [Completely Unsupervised Phoneme Recognition by Adversarially Learning Mapping Relationships from Audio Embeddings](https://mlsanity.com/paper/1804.00316/) (2018)
- [Unsupervised Speech Recognition via Segmental Empirical Output Distribution Matching](https://mlsanity.com/paper/1812.09323/) (2018)
- [Unsupervised Text-to-Speech Synthesis by Unsupervised Automatic Speech Recognition](https://mlsanity.com/paper/2203.15796/) (2022)
- [Low-Resource Speech-to-Text Translation](https://mlsanity.com/paper/1803.09164/) (2018)
- [Simple and Effective Unsupervised Speech Synthesis](https://mlsanity.com/paper/2204.02524/) (2022)
- [Towards speech-to-text translation without speech recognition](https://mlsanity.com/paper/1702.03856/) (2017)
- [Analyzing the Robustness of Unsupervised Speech Recognition](https://mlsanity.com/paper/2110.03509/) (2021)
- [Towards Unsupervised Speech-to-Text Translation](https://mlsanity.com/paper/1811.01307/) (2018)

## Built on by (most cited first, 12 of 12)

- [UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Units](https://mlsanity.com/paper/2212.08055/) (2022)
- [Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalization](https://mlsanity.com/paper/2305.11095/) (2023)
- [BLASER: A Text-Free Speech-to-Speech Translation Evaluation Metric](https://mlsanity.com/paper/2212.08486/) (2022)
- [Translatotron 3: Speech to Speech Translation with Monolingual Data](https://mlsanity.com/paper/2305.17547/) (2023)
- [Joint Pre-Training with Speech and Bilingual Text for Direct Speech to Speech Translation](https://mlsanity.com/paper/2210.17027/) (2022)
- [End-to-End Speech-to-Text Translation: A Survey](https://mlsanity.com/paper/2312.01053/) (2023)
- [Improving Cascaded Unsupervised Speech Translation with Denoising Back-translation](https://mlsanity.com/paper/2305.07455/) (2023)
- [Unsupervised Speech Recognition with N-Skipgram and Positional Unigram Matching](https://mlsanity.com/paper/2310.02382/) (2023)
- [Textless Speech-to-Speech Translation With Limited Parallel Data](https://mlsanity.com/paper/2305.15405/) (2023)
- [Towards Unsupervised Speech Recognition at the Syllable-Level](https://mlsanity.com/paper/2510.03639/) (2025)
- [SimulTron: On-Device Simultaneous Speech to Speech Translation](https://mlsanity.com/paper/2406.02133/) (2024)
- [RosettaSpeech: Zero-Shot Speech-to-Speech Translation without Parallel Speech](https://mlsanity.com/paper/2511.20974/) (2025)

## Abstract (arXiv)

The amount of labeled data to train models for speech tasks is limited for most languages, however, the data scarcity is exacerbated for speech translation which requires labeled data covering two different languages. To address this issue, we study a simple and effective approach to build speech translation systems without labeled data by leveraging recent advances in unsupervised speech recognition, machine translation and speech synthesis, either in a pipeline approach, or to generate pseudo-labels for training end-to-end speech translation models. Furthermore, we present an unsupervised domain adaptation technique for pre-trained speech models which improves the performance of downstream unsupervised speech recognition, especially for low-resource settings. Experiments show that unsupervised speech-to-text translation outperforms the previous unsupervised state of the art by 3.2 BLEU on the Libri-Trans benchmark, on CoVoST 2, our best systems outperform the best supervised end-to-end models (without pre-training) from only two years ago by an average of 5.0 BLEU over five X-En directions. We also report competitive results on MuST-C and CVSS benchmarks.