mlsanityData to 7 Oct 2026

Translatotron 2: High-quality direct speech-to-speech translation with voice preservation

arXiv 2107.08661 ↗First posted 19 Jul 2021cs.CL cs.LG cs.SDcited by 69

Not on the map. Why?

Abstract (arXiv)

We present Translatotron 2, a neural direct speech-to-speech translation model that can be trained end-to-end. Translatotron 2 consists of a speech encoder, a linguistic decoder, an acoustic synthesizer, and a single attention module that connects them together. Experimental results on three datasets consistently show that Translatotron 2 outperforms the original Translatotron by a large margin on both translation quality (up to +15.5 BLEU) and speech generation quality, and approaches the same of cascade systems. In addition, we propose a simple method for preserving speakers' voices from the source speech to the translation speech in a different language. Unlike existing approaches, the proposed method is able to preserve each speaker's voice on speaker turns without requiring for speaker segmentation. Furthermore, compared to existing approaches, it better preserves speaker's privacy and mitigates potential misuse of voice cloning for creating spoofing audio artifacts.

Key facts. Cited by 69 papers in the map (data through 7 Oct 2026). Builds on 43 of its 66 references that are in the map. First posted on arXiv 19 Jul 2021.

Builds on

43 of 66 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  4. 2017Neural Discrete Representation Learningcited by 4,650
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  6. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  7. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  8. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  9. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  10. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  11. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  12. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
31 more
  1. 2017Generalized End-to-End Loss for Speaker Verificationcited by 290
  2. 2018Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesiscited by 280
  3. 2018Efficient Neural Audio Synthesiscited by 218
  4. 2017Parallel WaveNet: Fast High-Fidelity Speech Synthesiscited by 214
  5. 2019ASVspoof 2019: Future Horizons in Spoofed and Fake Audio Detectioncited by 173
  6. 2021Direct speech-to-speech translation with discrete unitscited by 148
  7. 2020Improved Noisy Student Training for Automatic Speech Recognitioncited by 139
  8. 2020Recent Developments on ESPnet Toolkit Boosted by Conformercited by 132
  9. 2017Sequence-to-Sequence Models Can Directly Translate Foreign Speechcited by 121
  10. 2019Direct speech-to-speech translation with a sequence-to-sequence modelcited by 118
  11. 2019Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modelingcited by 86
  12. 2019Recognizing long-form speech using streaming end-to-end modelscited by 77
  13. 2020Voice Conversion Challenge 2020: Intra-lingual semi-parallel and cross-lingual voice conversioncited by 75
  14. 2020CoVoST 2 and Massively Multilingual Speech-to-Text Translationcited by 73
  15. 2019Learning to Speak Fluently in a Foreign Language: Multilingual Speech Synthesis and Cross-Language Voice Cloningcited by 66
  16. 2020Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modelingcited by 65
  17. 2018Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translationcited by 57
  18. 2016Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activationscited by 54
  19. 2021PnG BERT: Augmented BERT on Phonemes and Graphemes for Neural TTScited by 52
  20. 2020Parallel Tacotron: Non-Autoregressive and Controllable TTScited by 49
  21. 2019Location-Relative Attention Mechanisms For Robust Long-Form Speech Synthesiscited by 44
  22. 2020UWSpeech: Speech to Speech Translation for Unwritten Languagescited by 43
  23. 2021Large-Scale Self- and Semi-Supervised Learning for Speech Translationcited by 36
  24. 2019Robust Sequence-to-Sequence Acoustic Modeling with Stepwise Monotonic Attention for Neural TTScited by 33
  25. 2021Parallel Tacotron 2: A Non-Autoregressive Neural TTS Model with Differentiable Duration Modelingcited by 31
  26. 2019Speech-to-speech Translation between Untranscribed Unknown Languagescited by 27
  27. 2017Speech recognition for medical conversationscited by 18
  28. 2019One-To-Many Multilingual End-to-end Speech Translationcited by 17
  29. 2020TTS-by-TTS: TTS-driven Data Augmentation for Fast and High-Quality Speech Synthesiscited by 15
  30. 2020SkinAugment: Auto-Encoding Speaker Conversions for Automatic Speech Translationcited by 13
  31. 2019Forward-Backward Decoding for Regularizing End-to-End TTScited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 10; 2023: 19; 2024: 13; 2025: 15; 2026: 12.

69 papers.

Sort:

By area:

  1. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  2. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 166
  3. 2025Step-Audio 2 Technical Reportcited by 101
  4. 2022CVSS Corpus and Massively Multilingual Speech-to-Speech Translationcited by 71
  5. 2022UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Unitscited by 60
  6. 2023Transformers in Speech Processing: A Surveycited by 38
  7. 2022SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translationscited by 35
  8. 2022Speech-to-Speech Translation For A Real-world Unwritten Languagecited by 25
  9. 2023EmphAssess : a Prosodic Benchmark on Assessing Emphasis Transfer in Speech-to-Speech Modelscited by 21
  10. 2022Leveraging unsupervised and weakly-supervised data to improve direct speech-to-speech translationcited by 20
  11. 2022WaveFit: An Iterative and Non-autoregressive Neural Vocoder based on Fixed-Point Iterationcited by 20
  12. 2024StreamSpeech: Simultaneous Speech-to-Speech Translation with Multi-task Learningcited by 20
38 more
  1. 2022Textless Direct Speech-to-Speech Translation with Discrete Speech Representationcited by 19
  2. 2023PolyVoice: Language Models for Speech to Speech Translationcited by 19
  3. 2025High-Fidelity Simultaneous Speech-To-Speech Translationcited by 15
  4. 2023AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representationcited by 14
  5. 2022Simple and Effective Unsupervised Speech Translationcited by 12
  6. 2023Translatotron 3: Speech to Speech Translation with Monolingual Datacited by 12
  7. 2023Textless Unit-to-Unit training for Many-to-Many Multilingual Speech-to-Speech Translationcited by 11
  8. 2024MSLM-S2ST: A Multitask Speech Language Model for Textless Speech-to-Speech Translation with Speaker Style Preservationcited by 11
  9. 2023A Holistic Cascade System, benchmark, and Human Evaluation Protocol for Expressive Speech-to-Speech Translationcited by 10
  10. 2023ESPnet-ST-v2: Multipurpose Spoken Language Translation Toolkitcited by 10
  11. 2024TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservationcited by 10
  12. 2024A Non-autoregressive Generation Framework for End-to-End Simultaneous Speech-to-Speech Translationcited by 10
  13. 2023DASpeech: Directed Acyclic Transformer for Fast and High-quality Speech-to-Speech Translationcited by 9
  14. 2024SeamlessExpressiveLM: Speech Language Model for Expressive Speech-to-Speech Translation with Chain-of-Thoughtcited by 9
  15. 2024Efficient Training for Multilingual Visual Speech Recognition: Pre-training with Discretized Visual Speech Representationcited by 8
  16. 2025PRIM: Towards Practical In-Image Multilingual Machine Translationcited by 7
  17. 2026Simultaneous Speech-to-Speech Translation Without Aligned Datacited by 7
  18. 2022Make More of Your Data: Minimal Effort Data Augmentation for Automatic Speech Recognition and Translationcited by 6
  19. 2025UniSS: Unified Expressive Speech-to-Speech Translation with Your Voicecited by 6
  20. 2023Duplex Diffusion Models Improve Speech-to-Speech Translationcited by 5
  21. 2023Towards cross-language prosody transfer for dialogcited by 5
  22. 2023Multilingual Speech-to-Speech Translation into Multiple Target Languagescited by 5
  23. 2024Can We Achieve High-quality Direct Speech-to-Speech Translation without Parallel Speech Data?cited by 4
  24. 2023Textless Speech-to-Speech Translation With Limited Parallel Datacited by 3
  25. 2025SimulS2S-LLM: Unlocking Simultaneous Inference of Speech LLMs for Speech-to-Speech Translationcited by 3
  26. 2025Scheduled Interleaved Speech-Text Training for Speech-to-Speech Translation with LLMscited by 3
  27. 2026Scaling Open Discrete Audio Foundation Models with Interleaved Semantic, Acoustic, and Text Tokenscited by 3
  28. 2023Learning When to Speak: Latency and Quality Trade-offs for Simultaneous Speech-to-Speech Translation with Offline Modelscited by 2
  29. 2025Speech Vecalign: an Embedding-based Method for Aligning Parallel Speech Documentscited by 2
  30. 2022Direct Speech-to-speech Translation without Textual Annotation using Bottleneck Featurescited by 1
  31. 2023DiffS2UT: A Semantic Preserving Diffusion Model for Textless Direct Speech-to-Speech Translationcited by 1
  32. 2024Direct Punjabi to English speech translation using discrete unitscited by 1
  33. 2024SimulTron: On-Device Simultaneous Speech to Speech Translationcited by 1
  34. 2024CTC-based Non-autoregressive Textless Speech-to-Speech Translationcited by 1
  35. 2025Leveraging Unit Language Guidance to Advance Speech Modeling in Textless Speech-to-Speech Translationcited by 1
  36. 2025StressTransfer: Stress-Aware Speech-to-Speech Translation with Emphasis Preservationcited by 1
  37. 2026STEB: A Speech-to-Speech Translation Expressiveness Benchmark for Evaluating Beyond Translation Fidelitycited by 1
  38. 2026Is Prosody Lost in Translation? Fine-Grained Cross-Lingual Prosody Similarity Across Languagescited by 1

50 most cited of 69 All 69 (JSON)