mlsanityData to 7 Oct 2026

UWSpeech: Speech to Speech Translation for Unwritten Languages

arXiv 2006.07926 ↗First posted 14 Jun 2020eess.AS cs.CLcited by 43

Not on the map. Why?

Abstract (arXiv)

Existing speech to speech translation systems heavily rely on the text of target language: they usually translate source language either to target text and then synthesize target speech from text, or directly to target speech with target text for auxiliary training. However, those methods cannot be applied to unwritten target languages, which have no written text or phoneme available. In this paper, we develop a translation system for unwritten languages, named as UWSpeech, which converts target unwritten speech into discrete tokens with a converter, and then translates source-language speech into target discrete tokens with a translator, and finally synthesizes target speech from target discrete tokens with an inverter. We propose a method called XL-VAE, which enhances vector quantized variational autoencoder (VQ-VAE) with cross-lingual (XL) speech recognition, to train the converter and inverter of UWSpeech jointly. Experiments on Fisher Spanish-English conversation translation dataset show that UWSpeech outperforms direct translation and VQ-VAE baseline by about 16 and 10 BLEU points respectively, which demonstrate the advantages and potentials of UWSpeech.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 10; 2023: 12; 2024: 7; 2025: 5; 2026: 1.

43 papers.

Sort:

By area:

  1. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
  2. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  3. 2021A Survey on Neural Speech Synthesiscited by 182
  4. 2021Direct speech-to-speech translation with discrete unitscited by 148
  5. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  6. 2022CVSS Corpus and Massively Multilingual Speech-to-Speech Translationcited by 71
  7. 2021Translatotron 2: High-quality direct speech-to-speech translation with voice preservationcited by 69
  8. 2022UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Unitscited by 60
  9. 2022Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentationcited by 43
  10. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  11. 2022TranSpeech: Speech-to-Speech Translation With Bilateral Perturbationcited by 34
  12. 2022Speech-to-Speech Translation For A Real-world Unwritten Languagecited by 25
31 more
  1. 2021Variable-rate discrete representation learningcited by 24
  2. 2022Leveraging unsupervised and weakly-supervised data to improve direct speech-to-speech translationcited by 20
  3. 2022Textless Direct Speech-to-Speech Translation with Discrete Speech Representationcited by 19
  4. 2023PolyVoice: Language Models for Speech to Speech Translationcited by 19
  5. 2023DUB: Discrete Unit Back-translation for Speech Translationcited by 18
  6. 2021A Survey on Low-Resource Neural Machine Translationcited by 16
  7. 2022Leveraging Pseudo-labeled Data to Improve Direct Speech-to-Speech Translationcited by 12
  8. 2023Translatotron 3: Speech to Speech Translation with Monolingual Datacited by 12
  9. 2021Assessing Evaluation Metrics for Speech-to-Speech Translationcited by 10
  10. 2023ESPnet-ST-v2: Multipurpose Spoken Language Translation Toolkitcited by 10
  11. 2022Joint Pre-Training with Speech and Bilingual Text for Direct Speech to Speech Translationcited by 9
  12. 2023DASpeech: Directed Acyclic Transformer for Fast and High-quality Speech-to-Speech Translationcited by 9
  13. 2023AV-TranSpeech: Audio-Visual Robust Speech-to-Speech Translationcited by 6
  14. 2025UniSS: Unified Expressive Speech-to-Speech Translation with Your Voicecited by 6
  15. 2023Towards cross-language prosody transfer for dialogcited by 5
  16. 2023TransFace: Unit-Based Audio-Visual Speech Synthesizer for Talking Head Translationcited by 4
  17. 2024TMT: Tri-Modal Translation between Speech, Image, and Text by Processing Different Modalities as Different Languagescited by 4
  18. 2024Can We Achieve High-quality Direct Speech-to-Speech Translation without Parallel Speech Data?cited by 4
  19. 2025When End-to-End is Overkill: Rethinking Cascaded Speech-to-Text Translationcited by 4
  20. 2023Textless Speech-to-Speech Translation With Limited Parallel Datacited by 3
  21. 2023Learning Multilingual Expressive Speech Representation for Prosody Prediction without Parallel Datacited by 3
  22. 2024Do Discrete Self-Supervised Representations of Speech Capture Tone Distinctions?cited by 3
  23. 2024LLM-Ref: Enhancing Reference Handling in Technical Writing with Large Language Modelscited by 2
  24. 2022Dialogs Re-enacted Across Languagescited by 1
  25. 2024SimulTron: On-Device Simultaneous Speech to Speech Translationcited by 1
  26. 2025Leveraging Unit Language Guidance to Advance Speech Modeling in Textless Speech-to-Speech Translationcited by 1
  27. 2025MauBERT: Universal Phonetic Inductive Biases for Few-Shot Acoustic Units Discoverycited by 1
  28. 2023Enhancing Speech-to-Speech Translation with Multiple TTS Targetscited by 0
  29. 2024DiffNorm: Self-Supervised Normalization for Non-autoregressive Speech-to-speech Translationcited by 0
  30. 2024Phonology-Guided Speech-to-Speech Translation for African Languagescited by 0
  31. 2026PROST-LLM: Progressively Enhancing the Speech-to-Speech Translation Capability in LLMscited by 0