mlsanityData to 7 Oct 2026

Leveraging Pseudo-labeled Data to Improve Direct Speech-to-Speech Translation

arXiv 2205.08993 ↗First posted 18 May 2022cs.CL eess.AScited by 12

Not on the map. Why?

Abstract (arXiv)

Direct Speech-to-speech translation (S2ST) has drawn more and more attention recently. The task is very challenging due to data scarcity and complex speech-to-speech mapping. In this paper, we report our recent achievements in S2ST. Firstly, we build a S2ST Transformer baseline which outperforms the original Translatotron. Secondly, we utilize the external data by pseudo-labeling and obtain a new state-of-the-art result on the Fisher English-to-Spanish test set. Indeed, we exploit the pseudo data with a combination of popular techniques which are not trivial when applied to S2ST. Moreover, we evaluate our approach on both syntactically similar (Spanish-English) and distant (English-Chinese) language pairs. Our implementation is available at https://github.com/fengpeng-yue/speech-to-speech-translation.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 21 of its 28 references that are in the map. First posted on arXiv 18 May 2022.

Builds on

21 of 28 references.

References · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  3. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  4. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  5. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  6. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  7. 2017A Survey on Multi-Task Learningcited by 838
  8. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 487
  9. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  10. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  11. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 391
  12. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 364
9 more
  1. 2021Direct speech-to-speech translation with discrete unitscited by 148
  2. 2021WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkitcited by 123
  3. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  4. 2019Direct speech-to-speech translation with a sequence-to-sequence modelcited by 118
  5. 2020ESPnet-ST: All-in-One Speech Translation Toolkitcited by 82
  6. 2019End-to-End Speech Translation with Knowledge Distillationcited by 58
  7. 2018Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translationcited by 57
  8. 2020UWSpeech: Speech to Speech Translation for Unwritten Languagescited by 43
  9. 2020SkinAugment: Auto-Encoding Speaker Conversions for Automatic Speech Translationcited by 13

Built on by

Cited by year

Citing papers per year: 2022: 6; 2023: 6.

12 papers.

  1. 2022UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Unitscited by 60
  2. 2022Speech-to-Speech Translation For A Real-world Unwritten Languagecited by 25
  3. 2022Textless Direct Speech-to-Speech Translation with Discrete Speech Representationcited by 19
  4. 2023PolyVoice: Language Models for Speech to Speech Translationcited by 19
  5. 2023MixSpeech: Cross-Modality Self-Learning with Audio-Visual Stream Mixup for Visual Speech Translation and Recognitioncited by 14
  6. 2023Translatotron 3: Speech to Speech Translation with Monolingual Datacited by 12
  7. 2022Joint Pre-Training with Speech and Bilingual Text for Direct Speech to Speech Translationcited by 9
  8. 2023DASpeech: Directed Acyclic Transformer for Fast and High-quality Speech-to-Speech Translationcited by 9
  9. 2023Towards cross-language prosody transfer for dialogcited by 5
  10. 2022Joint Speech Transcription and Translation: Pseudo-Labeling with Out-of-Distribution Datacited by 3
  11. 2022Direct Speech-to-speech Translation without Textual Annotation using Bottleneck Featurescited by 1
  12. 2023Listen to Minority: Encrypted Traffic Classification for Class Imbalance with Contrastive Pre-Trainingcited by 1