mlsanityData to 14 Oct 2026

Joint Pre-Training with Speech and Bilingual Text for Direct Speech to Speech Translation

arXiv 2210.17027 ↗First posted 31 Oct 2022cs.SD cs.CL eess.AScited by 9

Not on the map. Why?

Abstract (arXiv)

Direct speech-to-speech translation (S2ST) is an attractive research topic with many advantages compared to cascaded S2ST. However, direct S2ST suffers from the data scarcity problem because the corpora from speech of the source language to speech of the target language are very rare. To address this issue, we propose in this paper a Speech2S model, which is jointly pre-trained with unpaired speech and bilingual text data for direct speech-to-speech translation tasks. By effectively leveraging the paired text data, Speech2S is capable of modeling the cross-lingual speech conversion from source to target language. We verify the performance of the proposed Speech2S on Europarl-ST and VoxPopuli datasets. Experimental results demonstrate that Speech2S gets an improvement of about 5 BLEU scores compared to encoder-only pre-training models, and achieves a competitive or even better performance than existing state-of-the-art models1.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 21 of its 29 references that are in the map. First posted on arXiv 31 Oct 2022.

Builds on

21 of 29 references.

References · 21

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  4. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  5. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  6. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 463
  7. 2021Direct speech-to-speech translation with discrete unitscited by 149
  8. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  9. 2019Direct speech-to-speech translation with a sequence-to-sequence modelcited by 118
  10. 2019Europarl-ST: A Multilingual Corpus For Speech Translation Of Parliamentary Debatescited by 114
  11. 2016Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translationcited by 113
  12. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
9 more
  1. 2022CVSS Corpus and Massively Multilingual Speech-to-Speech Translationcited by 71
  2. 2022Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentationcited by 44
  3. 2020UWSpeech: Speech to Speech Translation for Unwritten Languagescited by 43
  4. 2022SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-trainingcited by 40
  5. 2019Speech-to-speech Translation between Untranscribed Unknown Languagescited by 27
  6. 2019Self-Attention with Structural Position Representationscited by 21
  7. 2022Leveraging unsupervised and weakly-supervised data to improve direct speech-to-speech translationcited by 20
  8. 2022Leveraging Pseudo-labeled Data to Improve Direct Speech-to-Speech Translationcited by 12
  9. 2022Simple and Effective Unsupervised Speech Translationcited by 12

Built on by

Cited by year

Citing papers per year: 2023: 5; 2024: 4.

9 papers.

  1. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
  2. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 167
  3. 2023Translatotron 3: Speech to Speech Translation with Monolingual Datacited by 12
  4. 2024Low Frame-rate Speech Codec: a Codec Designed for Fast High-quality Speech LLM Training and Inferencecited by 11
  5. 2024TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservationcited by 10
  6. 2023DASpeech: Directed Acyclic Transformer for Fast and High-quality Speech-to-Speech Translationcited by 9
  7. 2023DiffS2UT: A Semantic Preserving Diffusion Model for Textless Direct Speech-to-Speech Translationcited by 1
  8. 2024SimulTron: On-Device Simultaneous Speech to Speech Translationcited by 1
  9. 2024TranSentence: Speech-to-speech Translation via Language-agnostic Sentence-level Speech Encoding without Language-parallel Datacited by 0