# Self-Supervised Representations Improve End-to-End Speech Translation

arXiv: https://arxiv.org/abs/2006.12124 · HTML: https://mlsanity.com/paper/2006.12124/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Jun 2020.

## Built on by (most cited first, 21 of 21)

- [VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://mlsanity.com/paper/2101.00390/) (2021)
- [Generative Spoken Language Modeling from Raw Audio](https://mlsanity.com/paper/2102.01192/) (2021)
- [ML-SUPERB: Multilingual Speech Universal PERformance Benchmark](https://mlsanity.com/paper/2305.10615/) (2023)
- [Cross-modal Contrastive Learning for Speech Translation](https://mlsanity.com/paper/2205.02444/) (2022)
- [What all do audio transformer models hear? Probing Acoustic Representations for Language Delivery and its Structure](https://mlsanity.com/paper/2101.00387/) (2021)
- [End-to-end Speech Translation via Cross-modal Progressive Training](https://mlsanity.com/paper/2104.10380/) (2021)
- [Recent Advances in Discrete Speech Tokens: A Review](https://mlsanity.com/paper/2502.06490/) (2025)
- [Large-Scale Self- and Semi-Supervised Learning for Speech Translation](https://mlsanity.com/paper/2104.06678/) (2021)
- [Probing self-supervised speech models for phonetic and phonemic information: a case study in aspiration](https://mlsanity.com/paper/2306.06232/) (2023)
- [Textless Speech Emotion Conversion using Discrete and Decomposed Representations](https://mlsanity.com/paper/2111.07402/) (2021)
- [Deep Transfer Learning for Automatic Speech Recognition: Towards Better Generalization](https://mlsanity.com/paper/2304.14535/) (2023)
- [WACO: Word-Aligned Contrastive Learning for Speech Translation](https://mlsanity.com/paper/2212.09359/) (2022)
- [Leveraging unsupervised and weakly-supervised data to improve direct speech-to-speech translation](https://mlsanity.com/paper/2203.13339/) (2022)
- [Recent Advances in Direct Speech-to-text Translation](https://mlsanity.com/paper/2306.11646/) (2023)
- [Combining Spectral and Self-Supervised Features for Low Resource Speech Recognition and Translation](https://mlsanity.com/paper/2204.02470/) (2022)
- [Findings of the 2023 ML-SUPERB Challenge: Pre-Training and Evaluation over More Languages and Beyond](https://mlsanity.com/paper/2310.05513/) (2023)
- [ivrit.ai: A Comprehensive Dataset of Hebrew Speech for AI Research and Development](https://mlsanity.com/paper/2307.08720/) (2023)
- [CoSTA: Code-Switched Speech Translation using Aligned Speech-Text Interleaving](https://mlsanity.com/paper/2406.10993/) (2024)
- [Bridging Speech and Textual Pre-trained Models with Unsupervised ASR](https://mlsanity.com/paper/2211.03025/) (2022)
- [Understanding Representation Learnability of Nonlinear Self-Supervised Learning](https://mlsanity.com/paper/2401.03214/) (2024)
- [Optimal Multi-Task Learning at Regularization Horizon for Speech Translation Task](https://mlsanity.com/paper/2509.09701/) (2025)

## Abstract (arXiv)

End-to-end speech-to-text translation can provide a simpler and smaller system but is facing the challenge of data scarcity. Pre-training methods can leverage unlabeled data and have been shown to be effective on data-scarce settings. In this work, we explore whether self-supervised pre-trained speech representations can benefit the speech translation task in both high- and low-resource settings, whether they can transfer well to other languages, and whether they can be effectively combined with other common methods that help improve low-resource end-to-end speech translation such as using a pre-trained high-resource speech recognition system. We demonstrate that self-supervised pre-trained features can consistently improve the translation performance, and cross-lingual transfer allows to extend to a variety of languages without or with little tuning.