mlsanityData to 14 Oct 2026

Speech-to-Speech Translation For A Real-world Unwritten Language

arXiv 2211.06474 ↗First posted 11 Nov 2022cs.CL cs.SD eess.AScited by 25

Not on the map. Why?

Abstract (arXiv)

We study speech-to-speech translation (S2ST) that translates speech from one language into another language and focuses on building systems to support languages without standard text writing systems. We use English-Taiwanese Hokkien as a case study, and present an end-to-end solution from training data collection, modeling choices to benchmark dataset release. First, we present efforts on creating human annotated data, automatically mining data from large unlabeled speech datasets, and adopting pseudo-labeling to produce weakly supervised data. On the modeling, we take advantage of recent advances in applying self-supervised discrete representations as target for prediction in S2ST and show the effectiveness of leveraging additional text supervision from Mandarin, a language similar to Hokkien, in model training. Finally, we release an S2ST benchmark set to facilitate future research in this field. The demo can be found at https://huggingface.co/spaces/facebook/Hokkien_Translation .

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Builds on 37 of its 60 references that are in the map. First posted on arXiv 11 Nov 2022.

Builds on

37 of 60 references.

References · 37

  1. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  4. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  6. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  7. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  8. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  9. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  10. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  11. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  12. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
25 more
  1. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 463
  2. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 439
  3. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  4. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  5. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  6. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  7. 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 226
  8. 2021R-Drop: Regularized Dropout for Neural Networkscited by 202
  9. 2021WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognitioncited by 197
  10. 2019CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEBcited by 177
  11. 2021Direct speech-to-speech translation with discrete unitscited by 149
  12. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  13. 2019Direct speech-to-speech translation with a sequence-to-sequence modelcited by 118
  14. 2019Europarl-ST: A Multilingual Corpus For Speech Translation Of Parliamentary Debatescited by 114
  15. 2016Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translationcited by 113
  16. 2021Translatotron 2: High-quality direct speech-to-speech translation with voice preservationcited by 71
  17. 2022CVSS Corpus and Massively Multilingual Speech-to-Speech Translationcited by 71
  18. 2022Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentationcited by 44
  19. 2020UWSpeech: Speech to Speech Translation for Unwritten Languagescited by 43
  20. 2022SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translationscited by 35
  21. 2022TranSpeech: Speech-to-Speech Translation With Bilateral Perturbationcited by 34
  22. 2019Speech-to-speech Translation between Untranscribed Unknown Languagescited by 27
  23. 2022Leveraging unsupervised and weakly-supervised data to improve direct speech-to-speech translationcited by 20
  24. 2022T-Modules: Translation Modules for Zero-Shot Cross-Modal Machine Translationcited by 16
  25. 2022Leveraging Pseudo-labeled Data to Improve Direct Speech-to-Speech Translationcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 8; 2024: 7; 2025: 5; 2026: 4.

25 papers.

Sort:

By area:

  1. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 43
  2. 2023Toward Joint Language Modeling for Speech Units and Textcited by 28
  3. 2024Align-SLM: Textless Spoken Language Models with Reinforcement Learning from AI Feedbackcited by 23
  4. 2023SpeechGen: Unlocking the Generative Power of Speech Language Models with Promptscited by 21
  5. 2022BLASER: A Text-Free Speech-to-Speech Translation Evaluation Metriccited by 16
  6. 2023ESPnet-ST-v2: Multipurpose Spoken Language Translation Toolkitcited by 10
  7. 2023DASpeech: Directed Acyclic Transformer for Fast and High-quality Speech-to-Speech Translationcited by 9
  8. 2023Evaluating Self-supervised Speech Models on a Taiwanese Hokkien Corpuscited by 7
  9. 2023AV-TranSpeech: Audio-Visual Robust Speech-to-Speech Translationcited by 6
  10. 2024Textless Acoustic Model with Self-Supervised Distillation for Noise-Robust Expressive Speech-to-Speech Translationcited by 5
  11. 2024Can We Achieve High-quality Direct Speech-to-Speech Translation without Parallel Speech Data?cited by 4
  12. 2023HK-LegiCoST: Leveraging Non-Verbatim Transcripts for Speech Translationcited by 3
13 more
  1. 2024Do Discrete Self-Supervised Representations of Speech Capture Tone Distinctions?cited by 3
  2. 2025CLiFT-ASR: A Cross-Lingual Fine-Tuning Framework for Low-Resource Taiwanese Hokkien Speech Recognitioncited by 2
  3. 2023DiffS2UT: A Semantic Preserving Diffusion Model for Textless Direct Speech-to-Speech Translationcited by 1
  4. 2024Direct Punjabi to English speech translation using discrete unitscited by 1
  5. 2024CTC-based Non-autoregressive Textless Speech-to-Speech Translationcited by 1
  6. 2025SiniticMTError: A Machine Translation Dataset with Error Annotations for Sinitic Languagescited by 1
  7. 2025MauBERT: Universal Phonetic Inductive Biases for Few-Shot Acoustic Units Discoverycited by 1
  8. 2026Towards Comprehensive Semantic Speech Embeddings for Chinese Dialectscited by 1
  9. 2026TaigiSpeech: A Low-Resource Real-World Speech Intent Dataset and Preliminary Results with Scalable Data Mining In-the-Wildcited by 1
  10. 2024DiffNorm: Self-Supervised Normalization for Non-autoregressive Speech-to-speech Translationcited by 0
  11. 2025Speech to Speech Translation with Translatotron: A State of the Art Reviewcited by 0
  12. 2026Evaluating and Preserving Lexical Stress in English-to-Chinese Speech-to-Speech Translationcited by 0
  13. 2026T-SANDHI: Tone Sandhi-aware Adaptive Network with Decoupled Hybrid Injection for Low-resource Taiwanese Hokkien Speech Recognitioncited by 0