# SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations

arXiv: https://arxiv.org/abs/2211.04508 · HTML: https://mlsanity.com/paper/2211.04508/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 55 references that are in the map.
- First posted on arXiv 8 Nov 2022.

## Builds on (most cited first, 30 of 42)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [Language-agnostic BERT Sentence Embedding](https://mlsanity.com/paper/2007.01852/) (2020)
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)
- [Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation](https://mlsanity.com/paper/2004.09813/) (2020)
- [Beyond English-Centric Multilingual Machine Translation](https://mlsanity.com/paper/2010.11125/) (2020)
- [The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation](https://mlsanity.com/paper/2106.03193/) (2021)
- [FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech](https://mlsanity.com/paper/2205.12446/) (2022)
- [VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://mlsanity.com/paper/2101.00390/) (2021)
- [Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond](https://mlsanity.com/paper/1812.10464/) (2018)
- [BASE Layers: Simplifying Training of Large, Sparse Models](https://mlsanity.com/paper/2103.16716/) (2021)
- [Speech Resynthesis from Discrete Disentangled Self-Supervised Representations](https://mlsanity.com/paper/2104.00355/) (2021)
- [Multilingual Universal Sentence Encoder for Semantic Retrieval](https://mlsanity.com/paper/1907.04307/) (2019)
- [Real Time Speech Enhancement in the Waveform Domain](https://mlsanity.com/paper/2006.12847/) (2020)
- [DeepNet: Scaling Transformers to 1,000 Layers](https://mlsanity.com/paper/2203.00555/) (2022)
- [CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEB](https://mlsanity.com/paper/1911.04944/) (2019)
- [Direct speech-to-speech translation with discrete units](https://mlsanity.com/paper/2107.05604/) (2021)
- [Textless Speech-to-Speech Translation on Real Data](https://mlsanity.com/paper/2112.08352/) (2021)
- [Direct speech-to-speech translation with a sequence-to-sequence model](https://mlsanity.com/paper/1904.06037/) (2019)
- [Europarl-ST: A Multilingual Corpus For Speech Translation Of Parliamentary Debates](https://mlsanity.com/paper/1911.03167/) (2019)
- [The Multilingual TEDx Corpus for Speech Recognition and Translation](https://mlsanity.com/paper/2102.01757/) (2021)
- [Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings](https://mlsanity.com/paper/1811.01136/) (2018)
- [Translatotron 2: High-quality direct speech-to-speech translation with voice preservation](https://mlsanity.com/paper/2107.08661/) (2021)
- [CVSS Corpus and Massively Multilingual Speech-to-Speech Translation](https://mlsanity.com/paper/2201.03713/) (2022)
- [Improving Multilingual Sentence Embedding using Bi-directional Dual Encoder with Additive Margin Softmax](https://mlsanity.com/paper/1902.08564/) (2019)

## Built on by (most cited first, 30 of 35)

- [Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling](https://mlsanity.com/paper/2303.03926/) (2023)
- [SONAR: Sentence-Level Multimodal and Language-Agnostic Representations](https://mlsanity.com/paper/2308.11466/) (2023)
- [ML-SUPERB: Multilingual Speech Universal PERformance Benchmark](https://mlsanity.com/paper/2305.10615/) (2023)
- [mHuBERT-147: A Compact Multilingual HuBERT Model](https://mlsanity.com/paper/2406.06371/) (2024)
- [Speech-to-Speech Translation For A Real-world Unwritten Language](https://mlsanity.com/paper/2211.06474/) (2022)
- [AfriHuBERT: A self-supervised speech representation model for African languages](https://mlsanity.com/paper/2409.20201/) (2024)
- [BLASER: A Text-Free Speech-to-Speech Translation Evaluation Metric](https://mlsanity.com/paper/2212.08486/) (2022)
- [Textless Unit-to-Unit training for Many-to-Many Multilingual Speech-to-Speech Translation](https://mlsanity.com/paper/2308.01831/) (2023)
- [MSLM-S2ST: A Multitask Speech Language Model for Textless Speech-to-Speech Translation with Speaker Style Preservation](https://mlsanity.com/paper/2403.12408/) (2024)
- [Findings of the 2023 ML-SUPERB Challenge: Pre-Training and Evaluation over More Languages and Beyond](https://mlsanity.com/paper/2310.05513/) (2023)
- [TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservation](https://mlsanity.com/paper/2405.17809/) (2024)
- [SeamlessExpressiveLM: Speech Language Model for Expressive Speech-to-Speech Translation with Chain-of-Thought](https://mlsanity.com/paper/2405.20410/) (2024)
- [AV-TranSpeech: Audio-Visual Robust Speech-to-Speech Translation](https://mlsanity.com/paper/2305.15403/) (2023)
- [Multilingual Speech-to-Speech Translation into Multiple Target Languages](https://mlsanity.com/paper/2307.08655/) (2023)
- [Transforming LLMs into Cross-modal and Cross-lingual Retrieval Systems](https://mlsanity.com/paper/2404.01616/) (2024)
- [Towards Building Large Scale Datasets and State-of-the-Art Automatic Speech Translation Systems for 14 Indian Languages](https://mlsanity.com/paper/2411.04699/) (2024)
- [Spatial Speech Translation: Translating Across Space With Binaural Hearables](https://mlsanity.com/paper/2504.18715/) (2025)
- [Modular Speech-to-Text Translation for Zero-Shot Cross-Modal Transfer](https://mlsanity.com/paper/2310.03724/) (2023)
- [Enhancing expressivity transfer in textless speech-to-speech translation](https://mlsanity.com/paper/2310.07279/) (2023)
- [A Case Study on Filtering for End-to-End Speech Translation](https://mlsanity.com/paper/2402.01945/) (2024)
- [Speech Vecalign: an Embedding-based Method for Aligning Parallel Speech Documents](https://mlsanity.com/paper/2509.18360/) (2025)
- [DiffS2UT: A Semantic Preserving Diffusion Model for Textless Direct Speech-to-Speech Translation](https://mlsanity.com/paper/2310.17570/) (2023)
- [Direct Punjabi to English speech translation using discrete units](https://mlsanity.com/paper/2402.15967/) (2024)
- [Dub-S2ST: Textless Speech-to-Speech Translation for Seamless Dubbing](https://mlsanity.com/paper/2505.20899/) (2025)
- [End-to-End Speech Translation for Low-Resource Languages Using Weakly Labeled Data](https://mlsanity.com/paper/2506.16251/) (2025)
- [STEB: A Speech-to-Speech Translation Expressiveness Benchmark for Evaluating Beyond Translation Fidelity](https://mlsanity.com/paper/2606.25529/) (2026)
- [TranSentence: Speech-to-speech Translation via Language-agnostic Sentence-level Speech Encoding without Language-parallel Data](https://mlsanity.com/paper/2401.12992/) (2024)
- [Diffusion Synthesizer for Efficient Multilingual Speech to Speech Translation](https://mlsanity.com/paper/2406.10223/) (2024)
- [Analyzing Speech Unit Selection for Textless Speech-to-Speech Translation](https://mlsanity.com/paper/2407.18332/) (2024)
- [Connecting Voices: LoReSpeech as a Low-Resource Speech Parallel Corpus](https://mlsanity.com/paper/2502.18215/) (2025)

## Abstract (arXiv)

We present SpeechMatrix, a large-scale multilingual corpus of speech-to-speech translations mined from real speech of European Parliament recordings. It contains speech alignments in 136 language pairs with a total of 418 thousand hours of speech. To evaluate the quality of this parallel speech, we train bilingual speech-to-speech translation models on mined data only and establish extensive baseline results on EuroParl-ST, VoxPopuli and FLEURS test sets. Enabled by the multilinguality of SpeechMatrix, we also explore multilingual speech-to-speech translation, a topic which was addressed by few other works. We also demonstrate that model pre-training and sparse scaling using Mixture-of-Experts bring large gains to translation performance. The mined data and models are freely available.