mlsanityData to 7 Oct 2026

SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations

arXiv 2211.04508 ↗First posted 8 Nov 2022cs.CL cs.SD eess.AScited by 35

Not on the map. Why?

Abstract (arXiv)

We present SpeechMatrix, a large-scale multilingual corpus of speech-to-speech translations mined from real speech of European Parliament recordings. It contains speech alignments in 136 language pairs with a total of 418 thousand hours of speech. To evaluate the quality of this parallel speech, we train bilingual speech-to-speech translation models on mined data only and establish extensive baseline results on EuroParl-ST, VoxPopuli and FLEURS test sets. Enabled by the multilinguality of SpeechMatrix, we also explore multilingual speech-to-speech translation, a topic which was addressed by few other works. We also demonstrate that model pre-training and sparse scaling using Mixture-of-Experts bring large gains to translation performance. The mined data and models are freely available.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 42 of its 55 references that are in the map. First posted on arXiv 8 Nov 2022.

Builds on

42 of 55 references.

References · 42

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  4. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  5. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  6. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  7. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  8. 2020Language-agnostic BERT Sentence Embeddingcited by 621
  9. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 591
  10. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  11. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  12. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
30 more
  1. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  2. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
  3. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  4. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  5. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  6. 2019Multilingual Universal Sentence Encoder for Semantic Retrievalcited by 192
  7. 2020Real Time Speech Enhancement in the Waveform Domaincited by 183
  8. 2019CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEBcited by 177
  9. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 176
  10. 2021Direct speech-to-speech translation with discrete unitscited by 148
  11. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  12. 2019Direct speech-to-speech translation with a sequence-to-sequence modelcited by 118
  13. 2019Europarl-ST: A Multilingual Corpus For Speech Translation Of Parliamentary Debatescited by 114
  14. 2021The Multilingual TEDx Corpus for Speech Recognition and Translationcited by 101
  15. 2018Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddingscited by 83
  16. 2022CVSS Corpus and Massively Multilingual Speech-to-Speech Translationcited by 71
  17. 2021Translatotron 2: High-quality direct speech-to-speech translation with voice preservationcited by 69
  18. 2019Improving Multilingual Sentence Embedding using Bi-directional Dual Encoder with Additive Margin Softmaxcited by 58
  19. 2020CoVoST: A Diverse Multilingual Speech-To-Text Translation Corpuscited by 57
  20. 2019CSS10: A Collection of Single Speaker Speech Datasets for 10 Languagescited by 54
  21. 2017Learning Joint Multilingual Sentence Representations with Neural Machine Translationcited by 52
  22. 2022Bitext Mining Using Distilled Sentence Representations for Low-Resource Languagescited by 48
  23. 2022Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentationcited by 43
  24. 2018Filtering and Mining Parallel Data in a Joint Multilingual Spacecited by 32
  25. 2022SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representationcited by 32
  26. 2019MaSS: A Large and Clean Multilingual Corpus of Sentence-aligned Spoken Utterances Extracted from the Biblecited by 24
  27. 2022Leveraging unsupervised and weakly-supervised data to improve direct speech-to-speech translationcited by 20
  28. 2022T-Modules: Translation Modules for Zero-Shot Cross-Modal Machine Translationcited by 16
  29. 2017An Empirical Analysis of NMT-Derived Interlingual Embeddings and their Use in Parallel Sentence Identificationcited by 9
  30. 2022Improving Speech-to-Speech Translation Through Unlabeled Textcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 10; 2024: 12; 2025: 5; 2026: 6.

35 papers.

Sort:

By area:

  1. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 166
  2. 2023SONAR: Sentence-Level Multimodal and Language-Agnostic Representationscited by 105
  3. 2023ML-SUPERB: Multilingual Speech Universal PERformance Benchmarkcited by 70
  4. 2024mHuBERT-147: A Compact Multilingual HuBERT Modelcited by 64
  5. 2022Speech-to-Speech Translation For A Real-world Unwritten Languagecited by 25
  6. 2024AfriHuBERT: A self-supervised speech representation model for African languagescited by 17
  7. 2022BLASER: A Text-Free Speech-to-Speech Translation Evaluation Metriccited by 16
  8. 2023Textless Unit-to-Unit training for Many-to-Many Multilingual Speech-to-Speech Translationcited by 11
  9. 2024MSLM-S2ST: A Multitask Speech Language Model for Textless Speech-to-Speech Translation with Speaker Style Preservationcited by 11
  10. 2023Findings of the 2023 ML-SUPERB Challenge: Pre-Training and Evaluation over More Languages and Beyondcited by 10
  11. 2024TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservationcited by 10
  12. 2024SeamlessExpressiveLM: Speech Language Model for Expressive Speech-to-Speech Translation with Chain-of-Thoughtcited by 9
23 more
  1. 2023AV-TranSpeech: Audio-Visual Robust Speech-to-Speech Translationcited by 6
  2. 2023Multilingual Speech-to-Speech Translation into Multiple Target Languagescited by 5
  3. 2024Transforming LLMs into Cross-modal and Cross-lingual Retrieval Systemscited by 5
  4. 2024Towards Building Large Scale Datasets and State-of-the-Art Automatic Speech Translation Systems for 14 Indian Languagescited by 4
  5. 2025Spatial Speech Translation: Translating Across Space With Binaural Hearablescited by 3
  6. 2023Modular Speech-to-Text Translation for Zero-Shot Cross-Modal Transfercited by 2
  7. 2023Enhancing expressivity transfer in textless speech-to-speech translationcited by 2
  8. 2024A Case Study on Filtering for End-to-End Speech Translationcited by 2
  9. 2025Speech Vecalign: an Embedding-based Method for Aligning Parallel Speech Documentscited by 2
  10. 2023DiffS2UT: A Semantic Preserving Diffusion Model for Textless Direct Speech-to-Speech Translationcited by 1
  11. 2024Direct Punjabi to English speech translation using discrete unitscited by 1
  12. 2025End-to-End Speech Translation for Low-Resource Languages Using Weakly Labeled Datacited by 1
  13. 2026STEB: A Speech-to-Speech Translation Expressiveness Benchmark for Evaluating Beyond Translation Fidelitycited by 1
  14. 2024TranSentence: Speech-to-speech Translation via Language-agnostic Sentence-level Speech Encoding without Language-parallel Datacited by 0
  15. 2024Diffusion Synthesizer for Efficient Multilingual Speech to Speech Translationcited by 0
  16. 2024Analyzing Speech Unit Selection for Textless Speech-to-Speech Translationcited by 0
  17. 2025Connecting Voices: LoReSpeech as a Low-Resource Speech Parallel Corpuscited by 0
  18. 2025Dub-S2ST: Textless Speech-to-Speech Translation for Seamless Dubbingcited by 0
  19. 2026FLiP: Towards understanding and interpreting multimodal multilingual sentence embeddingscited by 0
  20. 2026Leveraging Audio-LLMs to Filter Speech-to-Speech Training Datacited by 0
  21. 2026X-Translator: A Real-Time Multilingual Speaker-Aware Speech-to-Speech Translation Systemcited by 0
  22. 2026CVSS-X: A Multilingual Speech-to-Speech Translation Corpus for 28 Languagescited by 0
  23. 2026MGhana-ST: A Low-Resource Speech Translation Dataset for Ghanaian Languages and an Analysis of Multilingual Training Trade-offscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Breaking the Curse of Multilinguality in Many-to-Many Speech-to-Text Translation via a Resource-Aware Mixture of Speech Encoders
  2. 2025POTSA: A Cross-Lingual Speech Alignment Framework for Speech-to-Text Translation
  3. 2025PART: Progressive Alignment Representation Training for Multilingual Speech-To-Text with LLMs
  4. 2026LEMAS: Large A 150K-Hour Large-scale Extensible Multilingual Audio Suite with Generative Speech Models
  5. 2025Towards Fine-Grained Code-Switch Speech Translation with Semantic Space Alignment
  6. 2023Speech Translation with Large Language Models: An Industrial Practice
  7. 2025LatinX: Aligning a Multilingual TTS Model with Direct Preference Optimization
  8. 2025Bridging the Linguistic Divide: A Survey on Leveraging Large Language Models for Machine Translation