mlsanityData to 14 Oct 2026

Translatotron 3: Speech to Speech Translation with Monolingual Data

arXiv 2305.17547 ↗First posted 27 May 2023cs.CL cs.LG cs.SDcited by 12

Not on the map. Why?

Abstract (arXiv)

This paper presents Translatotron 3, a novel approach to unsupervised direct speech-to-speech translation from monolingual speech-text datasets by combining masked autoencoder, unsupervised embedding mapping, and back-translation. Experimental results in speech-to-speech translation tasks between Spanish and English show that Translatotron 3 outperforms a baseline cascade system, reporting 18.14 BLEU points improvement on the synthesized Unpaired-Conversational dataset. In contrast to supervised approaches that necessitate real paired data, or specialized modeling to replicate para-/non-linguistic information such as pauses, speaking rates, and speaker identity, Translatotron 3 showcases its capability to retain it. Audio samples can be found at http://google-research.github.io/lingvo-lab/translatotron3

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 49 of its 62 references that are in the map. First posted on arXiv 27 May 2023.

Builds on

49 of 62 references.

References · 49

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  4. 2017Neural Discrete Representation Learningcited by 4,712
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  6. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  7. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  8. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  9. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  10. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  11. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 788
  12. 2017Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networkscited by 707
37 more
  1. 2022High Fidelity Neural Audio Compressioncited by 687
  2. 2017Word Translation Without Parallel Datacited by 487
  3. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 347
  4. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  5. 2017Unsupervised Machine Translation Using Monolingual Corpora Onlycited by 273
  6. 2018Efficient Neural Audio Synthesiscited by 219
  7. 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 202
  8. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 180
  9. 2017Unsupervised Neural Machine Translationcited by 149
  10. 2021Direct speech-to-speech translation with discrete unitscited by 149
  11. 2018A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddingscited by 146
  12. 2019MOSNet: Deep Learning based Objective Assessment for Voice Conversioncited by 127
  13. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  14. 2019Direct speech-to-speech translation with a sequence-to-sequence modelcited by 118
  15. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  16. 2021Translatotron 2: High-quality direct speech-to-speech translation with voice preservationcited by 71
  17. 2022CVSS Corpus and Massively Multilingual Speech-to-Speech Translationcited by 71
  18. 2020Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modelingcited by 65
  19. 2022Unified Speech-Text Pre-training for Speech Translation and Recognitioncited by 64
  20. 2022UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Unitscited by 60
  21. 2018Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translationcited by 57
  22. 2021PnG BERT: Augmented BERT on Phonemes and Graphemes for Neural TTScited by 52
  23. 2022SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Datacited by 49
  24. 2020UWSpeech: Speech to Speech Translation for Unwritten Languagescited by 43
  25. 2022TranSpeech: Speech-to-Speech Translation With Bilateral Perturbationcited by 34
  26. 2019Speech-to-speech Translation between Untranscribed Unknown Languagescited by 27
  27. 2021LDNet: Unified Listener Dependent Modeling in MOS Prediction for Synthetic Speechcited by 24
  28. 2016AutoMOS: Learning a non-intrusive assessor of naturalness-of-speechcited by 22
  29. 2022WaveFit: An Iterative and Non-autoregressive Neural Vocoder based on Fixed-Point Iterationcited by 20
  30. 2022Textless Direct Speech-to-Speech Translation with Discrete Speech Representationcited by 19
  31. 2023DUB: Discrete Unit Back-translation for Speech Translationcited by 18
  32. 2022Leveraging Pseudo-labeled Data to Improve Direct Speech-to-Speech Translationcited by 12
  33. 2022Simple and Effective Unsupervised Speech Translationcited by 12
  34. 2022SQuId: Measuring Speech Naturalness in Many Languagescited by 10
  35. 2022Residual Adapters for Few-Shot Text-to-Speech Speaker Adaptationcited by 10
  36. 2022Joint Pre-Training with Speech and Bilingual Text for Direct Speech to Speech Translationcited by 9
  37. 2018Towards Unsupervised Speech-to-Text Translationcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 4; 2026: 2.

12 papers.

  1. 2024TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservationcited by 10
  2. 2024MedCycle: Unpaired Medical Report Generation via Cycle-Consistencycited by 2
  3. 2024Which Prosodic Features Matter Most for Pragmatics?cited by 2
  4. 2024SimulTron: On-Device Simultaneous Speech to Speech Translationcited by 1
  5. 2025A Unit-based System and Dataset for Expressive Direct Speech-to-Speech Translationcited by 1
  6. 2024TranSentence: Speech-to-speech Translation via Language-agnostic Sentence-level Speech Encoding without Language-parallel Datacited by 0
  7. 2024Diffusion Synthesizer for Efficient Multilingual Speech to Speech Translationcited by 0
  8. 2025Speech to Speech Translation with Translatotron: A State of the Art Reviewcited by 0
  9. 2025MAVFlow: Preserving Paralinguistic Elements with Conditional Flow Matching for Zero-Shot AV2AV Multilingual Translationcited by 0
  10. 2025RosettaSpeech: Zero-Shot Speech-to-Speech Translation without Parallel Speechcited by 0
  11. 2026MoVE: Translating Laughter and Tears via Mixture of Vocalization Experts in Speech-to-Speech Translationcited by 0
  12. 2026Benchmarking Speech-to-Speech Translation Modelscited by 0