mlsanityData to 7 Oct 2026

Textless Speech-to-Speech Translation on Real Data

arXiv 2112.08352 ↗First posted 15 Dec 2021cs.CL cs.AI cs.LGcited by 121

Not on the map. Why?

Abstract (arXiv)

We present a textless speech-to-speech translation (S2ST) system that can translate speech from one language into another language and can be built without the need of any text data. Different from existing work in the literature, we tackle the challenge in modeling multi-speaker target speech and train the systems with real-world S2ST data. The key to our approach is a self-supervised unit-based speech normalization technique, which finetunes a pre-trained speech encoder with paired audios from multiple speakers and a single reference speaker to reduce the variations due to accents, while preserving the lexical content. With only 10 minutes of paired data for speech normalization, we obtain on average 3.2 BLEU gain when training the S2ST model on the VoxPopuli S2ST dataset, compared to a baseline trained on un-normalized speech target. We also incorporate automatically mined S2ST data and show an additional 2.0 BLEU gain. To our knowledge, we are the first to establish a textless S2ST technique that can be trained with real-world data and works for multiple language pairs. Audio samples are available at https://facebookresearch.github.io/speech_translation/textless_s2st_real_data/index.html .

Key facts. Cited by 121 papers in the map (data through 7 Oct 2026). Builds on 33 of its 47 references that are in the map. First posted on arXiv 15 Dec 2021.

Builds on

33 of 47 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2017Neural Discrete Representation Learningcited by 4,650
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  5. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  6. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  7. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  8. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  9. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  10. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  11. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  12. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
21 more
  1. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 487
  2. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
  3. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  4. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  5. 2019CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEBcited by 177
  6. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 174
  7. 2021Direct speech-to-speech translation with discrete unitscited by 148
  8. 2019End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architecturescited by 120
  9. 2021Text-Free Prosody-Aware Generative Spoken Language Modelingcited by 119
  10. 2019Direct speech-to-speech translation with a sequence-to-sequence modelcited by 118
  11. 2019Europarl-ST: A Multilingual Corpus For Speech Translation Of Parliamentary Debatescited by 114
  12. 2016Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translationcited by 113
  13. 2017Deep Voice 2: Multi-Speaker Neural Text-to-Speechcited by 75
  14. 2020CoVoST 2 and Massively Multilingual Speech-to-Text Translationcited by 73
  15. 2019Effectiveness of self-supervised pre-training for speech recognitioncited by 72
  16. 2019ESPnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkitcited by 60
  17. 2019CSS10: A Collection of Single Speaker Speech Datasets for 10 Languagescited by 54
  18. 2020UWSpeech: Speech to Speech Translation for Unwritten Languagescited by 43
  19. 2019Speech-to-speech Translation between Untranscribed Unknown Languagescited by 27
  20. 2019MaSS: A Large and Clean Multilingual Corpus of Sentence-aligned Spoken Utterances Extracted from the Biblecited by 24
  21. 2021fairseq S^2: A Scalable and Integrable Speech Synthesis Toolkitcited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 25; 2023: 42; 2024: 32; 2025: 14; 2026: 8.

121 papers.

Sort:

By area:

  1. 2022High Fidelity Neural Audio Compressioncited by 673
  2. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  3. 2022AudioGen: Textually Guided Audio Generationcited by 292
  4. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  5. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 166
  6. 2024Spirit LM: Interleaved Spoken and Written Language Modelcited by 123
  7. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 116
  8. 2025Step-Audio 2 Technical Reportcited by 101
  9. 2023Textually Pretrained Speech Language Modelscited by 99
  10. 2023Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLMcited by 92
  11. 2022CVSS Corpus and Massively Multilingual Speech-to-Speech Translationcited by 71
  12. 2023ML-SUPERB: Multilingual Speech Universal PERformance Benchmarkcited by 70
38 more
  1. 2024mHuBERT-147: A Compact Multilingual HuBERT Modelcited by 64
  2. 2022UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Unitscited by 60
  3. 2024EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotionscited by 44
  4. 2022Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentationcited by 43
  5. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  6. 2022SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-trainingcited by 40
  7. 2024Let's Go Real Talk: Spoken Dialogue Model for Face-to-Face Conversationcited by 40
  8. 2022SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translationscited by 35
  9. 2022TranSpeech: Speech-to-Speech Translation With Bilateral Perturbationcited by 34
  10. 2022Self-supervised language learning from raw audio: Lessons from the Zero Resource Speech Challengecited by 34
  11. 2024Where Visual Speech Meets Language: VSP-LLM Framework for Efficient and Context-Aware Visual Speech Processingcited by 29
  12. 2022textless-lib: a Library for Textless Spoken Language Processingcited by 28
  13. 2023Make-A-Voice: Unified Voice Synthesis With Discrete Representationcited by 27
  14. 2023RepCodec: A Speech Representation Codec for Speech Tokenizationcited by 27
  15. 2023Toward Joint Language Modeling for Speech Units and Textcited by 27
  16. 2022Speech-to-Speech Translation For A Real-world Unwritten Languagecited by 25
  17. 2023EmphAssess : a Prosodic Benchmark on Assessing Emphasis Transfer in Speech-to-Speech Modelscited by 21
  18. 2022Leveraging unsupervised and weakly-supervised data to improve direct speech-to-speech translationcited by 20
  19. 2022Textless Direct Speech-to-Speech Translation with Discrete Speech Representationcited by 19
  20. 2023PolyVoice: Language Models for Speech to Speech Translationcited by 19
  21. 2023DUB: Discrete Unit Back-translation for Speech Translationcited by 18
  22. 2022Augmentation Invariant Discrete Representation for Generative Spoken Language Modelingcited by 17
  23. 2024AfriHuBERT: A self-supervised speech representation model for African languagescited by 17
  24. 2022T-Modules: Translation Modules for Zero-Shot Cross-Modal Machine Translationcited by 16
  25. 2022BLASER: A Text-Free Speech-to-Speech Translation Evaluation Metriccited by 16
  26. 2022DUAL: Discrete Spoken Unit Adaptive Learning for Textless Spoken Question Answeringcited by 15
  27. 2022Speaking Style Conversion in the Waveform Domain Using Discrete Self-Supervised Unitscited by 15
  28. 2023AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representationcited by 14
  29. 2024Paralinguistics-Aware Speech-Empowered Large Language Models for Natural Conversationcited by 14
  30. 2022Leveraging Pseudo-labeled Data to Improve Direct Speech-to-Speech Translationcited by 12
  31. 2023Translatotron 3: Speech to Speech Translation with Monolingual Datacited by 12
  32. 2023Zero Resource Code-switched Speech Benchmark Using Speech Utterance Pairs For Multiple Spoken Languagescited by 12
  33. 2024LLaST: Improved End-to-end Speech Translation System Leveraged by Large Language Modelscited by 12
  34. 2023Textless Unit-to-Unit training for Many-to-Many Multilingual Speech-to-Speech Translationcited by 11
  35. 2024MSLM-S2ST: A Multitask Speech Language Model for Textless Speech-to-Speech Translation with Speaker Style Preservationcited by 11
  36. 2023A Holistic Cascade System, benchmark, and Human Evaluation Protocol for Expressive Speech-to-Speech Translationcited by 10
  37. 2023ESPnet-ST-v2: Multipurpose Spoken Language Translation Toolkitcited by 10
  38. 2023Findings of the 2023 ML-SUPERB Challenge: Pre-Training and Evaluation over More Languages and Beyondcited by 10

50 most cited of 121 All 121 (JSON)