mlsanityData to 7 Oct 2026

CVSS Corpus and Massively Multilingual Speech-to-Speech Translation

arXiv 2201.03713 ↗First posted 11 Jan 2022cs.CL cs.SD eess.AScited by 71

Not on the map. Why?

Abstract (arXiv)

We introduce CVSS, a massively multilingual-to-English speech-to-speech translation (S2ST) corpus, covering sentence-level parallel S2ST pairs from 21 languages into English. CVSS is derived from the Common Voice speech corpus and the CoVoST 2 speech-to-text translation (ST) corpus, by synthesizing the translation text from CoVoST 2 into speech using state-of-the-art TTS systems. Two versions of translation speeches are provided: 1) CVSS-C: All the translation speeches are in a single high-quality canonical voice; 2) CVSS-T: The translation speeches are in voices transferred from the corresponding source speeches. In addition, CVSS provides normalized translation text which matches the pronunciation in the translation speech. On each version of CVSS, we built baseline multilingual direct S2ST models and cascade S2ST models, verifying the effectiveness of the corpus. To build strong cascade S2ST baselines, we trained an ST model on CoVoST 2, which outperforms the previous state-of-the-art trained on the corpus without extra data by 5.8 BLEU. Nevertheless, the performance of the direct S2ST models approaches the strong cascade baselines when trained from scratch, and with only 0.1 or 0.7 BLEU difference on ASR transcribed translation when initialized from matching ST models.

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). Builds on 33 of its 58 references that are in the map. First posted on arXiv 11 Jan 2022.

Builds on

33 of 58 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  4. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  5. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  6. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  7. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  8. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  9. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  10. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
  11. 2017Generalized End-to-End Loss for Speaker Verificationcited by 290
  12. 2018Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesiscited by 280
21 more
  1. 2018Efficient Neural Audio Synthesiscited by 218
  2. 2021Direct speech-to-speech translation with discrete unitscited by 148
  3. 2020Improved Noisy Student Training for Automatic Speech Recognitioncited by 139
  4. 2017Sequence-to-Sequence Models Can Directly Translate Foreign Speechcited by 121
  5. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  6. 2019Direct speech-to-speech translation with a sequence-to-sequence modelcited by 118
  7. 2019Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modelingcited by 86
  8. 2018Pre-training on high-resource speech recognition improves low-resource speech-to-text translationcited by 70
  9. 2021Translatotron 2: High-quality direct speech-to-speech translation with voice preservationcited by 69
  10. 2019Learning to Speak Fluently in a Foreign Language: Multilingual Speech Synthesis and Cross-Language Voice Cloningcited by 66
  11. 2020Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modelingcited by 65
  12. 2018Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translationcited by 57
  13. 2020CoVoST: A Diverse Multilingual Speech-To-Text Translation Corpuscited by 57
  14. 2021PnG BERT: Augmented BERT on Phonemes and Graphemes for Neural TTScited by 52
  15. 2019CN-CELEB: a challenging Chinese speaker recognition datasetcited by 47
  16. 2020UWSpeech: Speech to Speech Translation for Unwritten Languagescited by 43
  17. 2019Speech-to-speech Translation between Untranscribed Unknown Languagescited by 27
  18. 2019MaSS: A Large and Clean Multilingual Corpus of Sentence-aligned Spoken Utterances Extracted from the Biblecited by 24
  19. 2021Is "moby dick" a Whale or a Bird? Named Entities and Terminology in Speech Translationcited by 11
  20. 2021Dr-Vectors: Decision Residual Networks and an Improved Loss for Speaker Recognitioncited by 10
  21. 2021Direct Simultaneous Speech-to-Speech Translation with Variational Monotonic Multihead Attentioncited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 14; 2023: 19; 2024: 15; 2025: 12; 2026: 11.

71 papers.

Sort:

By area:

  1. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  2. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  3. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 166
  4. 2025Step-Audio 2 Technical Reportcited by 101
  5. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 88
  6. 2024Recent Advances in Speech Language Models: A Surveycited by 85
  7. 2022UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Unitscited by 60
  8. 2022NusaCrowd: Open Source Initiative for Indonesian NLP Resourcescited by 38
  9. 2022SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translationscited by 35
  10. 2022TranSpeech: Speech-to-Speech Translation With Bilateral Perturbationcited by 34
  11. 2022Speech-to-Speech Translation For A Real-world Unwritten Languagecited by 25
  12. 2022XTREME-S: Evaluating Cross-lingual Speech Representationscited by 22
38 more
  1. 2022Leveraging unsupervised and weakly-supervised data to improve direct speech-to-speech translationcited by 20
  2. 2022VideoDubber: Machine Translation with Speech-Aware Length Control for Video Dubbingcited by 20
  3. 2022Textless Direct Speech-to-Speech Translation with Discrete Speech Representationcited by 19
  4. 2023PolyVoice: Language Models for Speech to Speech Translationcited by 19
  5. 2025High-Fidelity Simultaneous Speech-To-Speech Translationcited by 15
  6. 2022Simple and Effective Unsupervised Speech Translationcited by 12
  7. 2023Translatotron 3: Speech to Speech Translation with Monolingual Datacited by 12
  8. 2023Textless Unit-to-Unit training for Many-to-Many Multilingual Speech-to-Speech Translationcited by 11
  9. 2023ESPnet-ST-v2: Multipurpose Spoken Language Translation Toolkitcited by 10
  10. 2024TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservationcited by 10
  11. 2022Joint Pre-Training with Speech and Bilingual Text for Direct Speech to Speech Translationcited by 9
  12. 2023DASpeech: Directed Acyclic Transformer for Fast and High-quality Speech-to-Speech Translationcited by 9
  13. 2024MSR-86K: An Evolving, Multilingual Corpus with 86,300 Hours of Transcribed Audio for Speech Recognition Researchcited by 9
  14. 2023AV-TranSpeech: Audio-Visual Robust Speech-to-Speech Translationcited by 6
  15. 2025UniSS: Unified Expressive Speech-to-Speech Translation with Your Voicecited by 6
  16. 2023Improving Cascaded Unsupervised Speech Translation with Denoising Back-translationcited by 5
  17. 2023Duplex Diffusion Models Improve Speech-to-Speech Translationcited by 5
  18. 2023Towards cross-language prosody transfer for dialogcited by 5
  19. 2023Multilingual Speech-to-Speech Translation into Multiple Target Languagescited by 5
  20. 2023Speech-to-Speech Translation with Discrete-Unit-Based Style Transfercited by 5
  21. 2025MMMG: a Comprehensive and Reliable Benchmark for Multitask Multimodal Generationcited by 5
  22. 2022A Textless Metric for Speech-to-Speech Comparisoncited by 4
  23. 2023TransFace: Unit-Based Audio-Visual Speech Synthesizer for Talking Head Translationcited by 4
  24. 2024Overcoming Non-monotonicity in Transducer-based Streaming Generationcited by 4
  25. 2023HK-LegiCoST: Leveraging Non-Verbatim Transcripts for Speech Translationcited by 3
  26. 2024FFSTC: Fongbe to French Speech Translation Corpuscited by 3
  27. 2025Universal Speech Token Learning via Low-Bitrate Neural Codec and Pretrained Representationscited by 3
  28. 2025Scheduled Interleaved Speech-Text Training for Speech-to-Speech Translation with LLMscited by 3
  29. 2026Scaling Open Discrete Audio Foundation Models with Interleaved Semantic, Acoustic, and Text Tokenscited by 3
  30. 2024Multi-modal Speech Transformer Decoders: When Do Multiple Modalities Improve Accuracy?cited by 2
  31. 2025Inclusivity of AI Speech in Healthcare: A Decade Look Backcited by 2
  32. 2022Dialogs Re-enacted Across Languagescited by 1
  33. 2024Direct Punjabi to English speech translation using discrete unitscited by 1
  34. 2024CrossVoice: Crosslingual Prosody Preserving Cascade-S2ST using Transfer Learningcited by 1
  35. 2025Miipher-2: A Universal Speech Restoration Model for Million-Hour Scale Data Restorationcited by 1
  36. 2025MLLM-based Speech Recognition: When and How is Multimodality Beneficial?cited by 1
  37. 2025MTP-S2UT: Enhancing Speech-to-Speech Translation Quality with Multi-token Predictioncited by 1
  38. 2023Enhancing Speech-to-Speech Translation with Multiple TTS Targetscited by 0

50 most cited of 71 All 71 (JSON)