mlsanityData to 14 Oct 2026

PolyVoice: Language Models for Speech to Speech Translation

arXiv 2306.02982 ↗First posted 5 Jun 2023cs.CL eess.AScited by 19

Not on the map. Why?

Abstract (arXiv)

We propose PolyVoice, a language model-based framework for speech-to-speech translation (S2ST) system. Our framework consists of two language models: a translation language model and a speech synthesis language model. We use discretized speech units, which are generated in a fully unsupervised way, and thus our framework can be used for unwritten languages. For the speech synthesis part, we adopt the existing VALL-E X approach and build a unit-based audio language model. This grants our framework the ability to preserve the voice characteristics and the speaking style of the original speech. We examine our system on Chinese rightarrow English and English rightarrow Spanish pairs. Experimental results show that our system can generate speech with high translation quality and audio quality. Speech samples are available at https://speechtranslation.github.io/polyvoice.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 30 of its 37 references that are in the map. First posted on arXiv 5 Jun 2023.

Builds on

30 of 37 references.

References · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  5. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  6. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  7. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  8. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 561
  9. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 490
  10. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  11. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speechcited by 447
  12. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 399
18 more
  1. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  2. 2018Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesiscited by 282
  3. 2021WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognitioncited by 197
  4. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 167
  5. 2021Direct speech-to-speech translation with discrete unitscited by 149
  6. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  7. 2019Direct speech-to-speech translation with a sequence-to-sequence modelcited by 118
  8. 2019Zero-Shot Multi-Speaker Text-To-Speech with State-of-the-art Neural Speaker Embeddingscited by 73
  9. 2021Translatotron 2: High-quality direct speech-to-speech translation with voice preservationcited by 71
  10. 2022CVSS Corpus and Massively Multilingual Speech-to-Speech Translationcited by 71
  11. 2020UWSpeech: Speech to Speech Translation for Unwritten Languagescited by 43
  12. 2019Speech-to-speech Translation between Untranscribed Unknown Languagescited by 27
  13. 2022Are discrete units necessary for Spoken Language Modeling?cited by 24
  14. 2023DUB: Discrete Unit Back-translation for Speech Translationcited by 18
  15. 2022Leveraging Pseudo-labeled Data to Improve Direct Speech-to-Speech Translationcited by 12
  16. 2019Cross-lingual Multi-speaker Text-to-speech Synthesis for Voice Cloning without Using Parallel Corpus for Unseen Speakerscited by 11
  17. 2022Cross-Lingual Text-to-Speech Using Multi-Task Learning and Speaker Classifier Joint Trainingcited by 7
  18. 2022CoBERT: Self-Supervised Speech Representation Learning Through Code Representation Learningcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 8; 2025: 4; 2026: 2.

19 papers.

Sort:

By area:

  1. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 122
  2. 2023SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Modelscited by 87
  3. 2023Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation taskscited by 72
  4. 2023Speech Translation with Large Language Models: An Industrial Practicecited by 28
  5. 2023RepCodec: A Speech Representation Codec for Speech Tokenizationcited by 27
  6. 2024Paralinguistics-Aware Speech-Empowered Large Language Models for Natural Conversationcited by 14
  7. 2024MSLM-S2ST: A Multitask Speech Language Model for Textless Speech-to-Speech Translation with Speaker Style Preservationcited by 11
  8. 2024TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservationcited by 10
  9. 2024SeamlessExpressiveLM: Speech Language Model for Expressive Speech-to-Speech Translation with Chain-of-Thoughtcited by 9
  10. 2025UniSS: Unified Expressive Speech-to-Speech Translation with Your Voicecited by 7
  11. 2024SpeechComposer: Unifying Multiple Speech Tasks with Prompt Compositioncited by 3
  12. 2024SynesLM: A Unified Approach for Audio-visual Speech Recognition and Translation via Language Model and Synthetic Datacited by 3
7 more
  1. 2025Spatial Speech Translation: Translating Across Space With Binaural Hearablescited by 3
  2. 2025Scheduled Interleaved Speech-Text Training for Speech-to-Speech Translation with LLMscited by 3
  3. 2024An Empirical Study of Speech Language Models for Prompt-Conditioned Speech Synthesiscited by 1
  4. 2023I2KD-SLU: An Intra-Inter Knowledge Distillation Framework for Zero-Shot Cross-Lingual Spoken Language Understandingcited by 0
  5. 2025Breaking Through the Spike: Spike Window Decoding for Accelerated and Precise Automatic Speech Recognitioncited by 0
  6. 2026Evaluating and Preserving Lexical Stress in English-to-Chinese Speech-to-Speech Translationcited by 0
  7. 2026DuraS2ST: Chain-of-Thought and Reinforcement Learning for Duration-Aligned Speech-to-Speech Translationcited by 0