mlsanityData to 7 Oct 2026

Can We Achieve High-quality Direct Speech-to-Speech Translation without Parallel Speech Data?

arXiv 2406.07289 ↗First posted 11 Jun 2024cs.CL cs.AI cs.SDcited by 4

Not on the map. Why?

Abstract (arXiv)

Recently proposed two-pass direct speech-to-speech translation (S2ST) models decompose the task into speech-to-text translation (S2TT) and text-to-speech (TTS) within an end-to-end model, yielding promising results. However, the training of these models still relies on parallel speech data, which is extremely challenging to collect. In contrast, S2TT and TTS have accumulated a large amount of data and pretrained models, which have not been fully utilized in the development of S2ST models. Inspired by this, in this paper, we first introduce a composite S2ST model named ComSpeech, which can seamlessly integrate any pretrained S2TT and TTS models into a direct S2ST model. Furthermore, to eliminate the reliance on parallel speech data, we propose a novel training method ComSpeech-ZS that solely utilizes S2TT and TTS data. It aligns representations in the latent space through contrastive learning, enabling the speech synthesis capability learned from the TTS data to generalize to S2ST in a zero-shot manner. Experimental results on the CVSS dataset show that when the parallel speech data is available, ComSpeech surpasses previous two-pass models like UnitY and Translatotron 2 in both translation quality and decoding speed. When there is no parallel speech data, ComSpeech-ZS lags behind name by only 0.7 ASR-BLEU and outperforms the cascaded models.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 31 of its 39 references that are in the map. First posted on arXiv 11 Jun 2024.

Builds on

31 of 39 references.

References · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  4. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  5. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  6. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  7. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  8. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  9. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 174
  10. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 166
  11. 2021Direct speech-to-speech translation with discrete unitscited by 148
  12. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
19 more
  1. 2019Direct speech-to-speech translation with a sequence-to-sequence modelcited by 118
  2. 2021Translatotron 2: High-quality direct speech-to-speech translation with voice preservationcited by 69
  3. 2022UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Unitscited by 60
  4. 2022STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translationcited by 57
  5. 2020Bridging the Modality Gap for Speech-to-Text Translationcited by 55
  6. 2021CTC-based Compression for Direct Speech Translationcited by 45
  7. 2020UWSpeech: Speech to Speech Translation for Unwritten Languagescited by 43
  8. 2022Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentationcited by 43
  9. 2022Speech-to-Speech Translation For A Real-world Unwritten Languagecited by 25
  10. 2023CMOT: Cross-modal Mixup via Optimal Transport for Speech Translationcited by 22
  11. 2022Leveraging unsupervised and weakly-supervised data to improve direct speech-to-speech translationcited by 20
  12. 2022T-Modules: Translation Modules for Zero-Shot Cross-Modal Machine Translationcited by 16
  13. 2023Understanding and Bridging the Modality Gap for Speech Translationcited by 15
  14. 2023Back Translation for Speech-to-text Translation Without Transcriptscited by 9
  15. 2022Improving Speech-to-Speech Translation Through Unlabeled Textcited by 8
  16. 2022Discrete Cross-Modal Alignment Enables Zero-Shot Speech Translationcited by 6
  17. 2021Zero-shot Speech Translationcited by 5
  18. 2023Duplex Diffusion Models Improve Speech-to-Speech Translationcited by 5
  19. 2023Modular Speech-to-Text Translation for Zero-Shot Cross-Modal Transfercited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025PRIM: Towards Practical In-Image Multilingual Machine Translationcited by 7
  2. 2025Speech to Speech Translation with Translatotron: A State of the Art Reviewcited by 0
  3. 2025RosettaSpeech: Zero-Shot Speech-to-Speech Translation without Parallel Speechcited by 0
  4. 2026From Flat Language Labels to Typological Priors: Structured Language Conditioning for Multilingual Speech-to-Speech Translationcited by 0