mlsanityData to 7 Oct 2026

UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Units

arXiv 2212.08055 ↗First posted 15 Dec 2022cs.CL cs.SD eess.AScited by 60

Not on the map. Why?

Abstract (arXiv)

Direct speech-to-speech translation (S2ST), in which all components can be optimized jointly, is advantageous over cascaded approaches to achieve fast inference with a simplified pipeline. We present a novel two-pass direct S2ST architecture, UnitY, which first generates textual representations and predicts discrete acoustic units subsequently. We enhance the model performance by subword prediction in the first-pass decoder, advanced two-pass decoder architecture design and search strategy, and better training regularization. To leverage large amounts of unlabeled text data, we pre-train the first-pass text decoder based on the self-supervised denoising auto-encoding task. Experimental evaluations on benchmark datasets at various data scales demonstrate that UnitY outperforms a single-pass speech-to-unit translation model by 2.5-4.2 ASR-BLEU with 2.83x decoding speed-up. We show that the proposed methods boost the performance even when predicting spectrogram in the second pass. However, predicting discrete units achieves 2.51x decoding speed-up compared to that case.

Key facts. Cited by 60 papers in the map (data through 7 Oct 2026). Builds on 71 of its 108 references that are in the map. First posted on arXiv 15 Dec 2022.

Builds on

71 of 108 references.

References · 71

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  3. 2017Neural Discrete Representation Learningcited by 4,650
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  5. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  7. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  8. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  9. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  10. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  11. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  12. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
38 more
  1. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  2. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  3. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  4. 2017Mixed Precision Trainingcited by 1,135
  5. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  6. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  7. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 591
  8. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  9. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  10. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
  11. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  12. 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
  13. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 253
  14. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  15. 2021R-Drop: Regularized Dropout for Neural Networkscited by 202
  16. 2019CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEBcited by 177
  17. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 174
  18. 2015End-to-End Attention-based Large Vocabulary Speech Recognitioncited by 148
  19. 2021Direct speech-to-speech translation with discrete unitscited by 148
  20. 2017Sequence-to-Sequence Models Can Directly Translate Foreign Speechcited by 121
  21. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  22. 2019Direct speech-to-speech translation with a sequence-to-sequence modelcited by 118
  23. 2020VoxLingua107: a Dataset for Spoken Language Recognitioncited by 118
  24. 2019Europarl-ST: A Multilingual Corpus For Speech Translation Of Parliamentary Debatescited by 114
  25. 2016Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translationcited by 113
  26. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  27. 2020Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translationcited by 104
  28. 2021The Multilingual TEDx Corpus for Speech Recognition and Translationcited by 101
  29. 2021SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Networkcited by 83
  30. 2022CVSS Corpus and Massively Multilingual Speech-to-Speech Translationcited by 71
  31. 2021Translatotron 2: High-quality direct speech-to-speech translation with voice preservationcited by 69
  32. 2018End-to-End Automatic Speech Translation of Audiobookscited by 65
  33. 2020Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modelingcited by 65
  34. 2022Unified Speech-Text Pre-training for Speech Translation and Recognitioncited by 64
  35. 2019Two-Pass End-to-End Speech Recognitioncited by 62
  36. 2019End-to-End Speech Translation with Knowledge Distillationcited by 58
  37. 2018Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translationcited by 57
  38. 2020Deliberation Model Based Two-Pass End-to-End Speech Recognitioncited by 57

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 15; 2024: 17; 2025: 19; 2026: 8.

60 papers.

Sort:

By area:

  1. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  2. 2024Autoregressive Diffusion Transformer for Text-to-Speech Synthesiscited by 34
  3. 2024DASB - Discrete Audio and Speech Benchmarkcited by 32
  4. 2023Intelligible Lip-to-Speech Synthesis with Speech Unitscited by 28
  5. 2024StreamSpeech: Simultaneous Speech-to-Speech Translation with Multi-task Learningcited by 20
  6. 2023DUB: Discrete Unit Back-translation for Speech Translationcited by 18
  7. 2022BLASER: A Text-Free Speech-to-Speech Translation Evaluation Metriccited by 16
  8. 2025High-Fidelity Simultaneous Speech-To-Speech Translationcited by 15
  9. 2023AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representationcited by 14
  10. 2024Speech is More Than Words: Do Speech-to-Text Translation Systems Leverage Prosody?cited by 13
  11. 2025Scaling Analysis of Interleaved Speech-Text Language Modelscited by 13
  12. 2023Translatotron 3: Speech to Speech Translation with Monolingual Datacited by 12
38 more
  1. 2023Textless Unit-to-Unit training for Many-to-Many Multilingual Speech-to-Speech Translationcited by 11
  2. 2023ESPnet-ST-v2: Multipurpose Spoken Language Translation Toolkitcited by 10
  3. 2024TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservationcited by 10
  4. 2024A Non-autoregressive Generation Framework for End-to-End Simultaneous Speech-to-Speech Translationcited by 10
  5. 2023Back Translation for Speech-to-text Translation Without Transcriptscited by 9
  6. 2023DASpeech: Directed Acyclic Transformer for Fast and High-quality Speech-to-Speech Translationcited by 9
  7. 2023Efficient Monotonic Multihead Attentioncited by 8
  8. 2023GSQA: An End-to-End Model for Generative Spoken Question Answeringcited by 8
  9. 2025Exploring In-Image Machine Translation with Real-World Backgroundcited by 7
  10. 2025PRIM: Towards Practical In-Image Multilingual Machine Translationcited by 7
  11. 2025UniSS: Unified Expressive Speech-to-Speech Translation with Your Voicecited by 6
  12. 2023Duplex Diffusion Models Improve Speech-to-Speech Translationcited by 5
  13. 2023Multilingual Speech-to-Speech Translation into Multiple Target Languagescited by 5
  14. 2024Textless Acoustic Model with Self-Supervised Distillation for Noise-Robust Expressive Speech-to-Speech Translationcited by 5
  15. 2023Towards Practical and Efficient Image-to-Speech Captioning with Vision-Language Pre-training and Multi-modal Tokenscited by 4
  16. 2024TMT: Tri-Modal Translation between Speech, Image, and Text by Processing Different Modalities as Different Languagescited by 4
  17. 2024Can We Achieve High-quality Direct Speech-to-Speech Translation without Parallel Speech Data?cited by 4
  18. 2023Cross-Modal Multi-Tasking for Speech-to-Text Translation via Hard Parameter Sharingcited by 3
  19. 2025Discrete Speech Unit Extraction via Independent Component Analysiscited by 3
  20. 2025Universal Speech Token Learning via Low-Bitrate Neural Codec and Pretrained Representationscited by 3
  21. 2025SimulS2S-LLM: Unlocking Simultaneous Inference of Speech LLMs for Speech-to-Speech Translationcited by 3
  22. 2025Scheduled Interleaved Speech-Text Training for Speech-to-Speech Translation with LLMscited by 3
  23. 2026Scaling Open Discrete Audio Foundation Models with Interleaved Semantic, Acoustic, and Text Tokenscited by 3
  24. 2023Learning When to Speak: Latency and Quality Trade-offs for Simultaneous Speech-to-Speech Translation with Offline Modelscited by 2
  25. 2025Advances in LLMs with Focus on Reasoning, Adaptability, Efficiency and Ethicscited by 2
  26. 2025Speech Vecalign: an Embedding-based Method for Aligning Parallel Speech Documentscited by 2
  27. 2024Direct Punjabi to English speech translation using discrete unitscited by 1
  28. 2024SimulTron: On-Device Simultaneous Speech to Speech Translationcited by 1
  29. 2024CTC-based Non-autoregressive Textless Speech-to-Speech Translationcited by 1
  30. 2025Leveraging Unit Language Guidance to Advance Speech Modeling in Textless Speech-to-Speech Translationcited by 1
  31. 2025End-to-End Speech Translation for Low-Resource Languages Using Weakly Labeled Datacited by 1
  32. 2025MTP-S2UT: Enhancing Speech-to-Speech Translation Quality with Multi-token Predictioncited by 1
  33. 2025StressTransfer: Stress-Aware Speech-to-Speech Translation with Emphasis Preservationcited by 1
  34. 2026STEB: A Speech-to-Speech Translation Expressiveness Benchmark for Evaluating Beyond Translation Fidelitycited by 1
  35. 2026Is Prosody Lost in Translation? Fine-Grained Cross-Lingual Prosody Similarity Across Languagescited by 1
  36. 2024TranSentence: Speech-to-speech Translation via Language-agnostic Sentence-level Speech Encoding without Language-parallel Datacited by 0
  37. 2024DiffNorm: Self-Supervised Normalization for Non-autoregressive Speech-to-speech Translationcited by 0
  38. 2024Diffusion Synthesizer for Efficient Multilingual Speech to Speech Translationcited by 0

50 most cited of 60 All 60 (JSON)