mlsanityData to 13 Oct 2026

Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling

arXiv 2303.03926 ↗First posted 7 Mar 2023cs.CL cs.AI cs.SDcited by 167

Not on the map. Why?

Cited by 27 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We propose a cross-lingual neural codec language model, VALL-E X, for cross-lingual speech synthesis. Specifically, we extend VALL-E and train a multi-lingual conditional codec language model to predict the acoustic token sequences of the target language speech by using both the source language speech and the target language text as prompts. VALL-E X inherits strong in-context learning capabilities and can be applied for zero-shot cross-lingual text-to-speech synthesis and zero-shot speech-to-speech translation tasks. Experimental results show that it can generate high-quality speech in the target language via just one speech utterance in the source language as a prompt while preserving the unseen speaker's voice, emotion, and acoustic environment. Moreover, VALL-E X effectively alleviates the foreign accent problems, which can be controlled by a language ID. Audio samples are available at urlhttps://aka.ms/vallex.

Key facts. Cited by 167 papers in the map (data through 13 Oct 2026). Builds on 40 of its 56 references that are in the map. First posted on arXiv 7 Mar 2023.

Builds on

40 of 56 references.

References · 40

  1. 2020Language Models are Few-Shot Learnerscited by 32,320
  2. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,313
  3. 2016WaveNet: A Generative Model for Raw Audiocited by 1,717
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,625
  5. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,348
  6. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,148
  7. 2018Self-Attention with Relative Position Representationscited by 1,116
  8. 2021SoundStream: An End-to-End Neural Audio Codeccited by 762
  9. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 745
  10. 2022High Fidelity Neural Audio Compressioncited by 680
  11. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 613
  12. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 464
28 more
  1. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 456
  2. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 393
  3. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 342
  4. 2022AudioGen: Textually Guided Audio Generationcited by 294
  5. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 278
  6. 2022Diffsound: Discrete Diffusion Model for Text-to-sound Generationcited by 257
  7. 2021WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognitioncited by 191
  8. 2022NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Qualitycited by 152
  9. 2021Direct speech-to-speech translation with discrete unitscited by 148
  10. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  11. 2019Direct speech-to-speech translation with a sequence-to-sequence modelcited by 118
  12. 2022CVSS Corpus and Massively Multilingual Speech-to-Speech Translationcited by 71
  13. 2021Translatotron 2: High-quality direct speech-to-speech translation with voice preservationcited by 70
  14. 2019Learning to Speak Fluently in a Foreign Language: Multilingual Speech Synthesis and Cross-Language Voice Cloningcited by 66
  15. 2022SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Datacited by 49
  16. 2022SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-trainingcited by 40
  17. 2021Deep Learning Based Assessment of Synthetic Speech Naturalnesscited by 35
  18. 2022SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translationscited by 35
  19. 2022TranSpeech: Speech-to-Speech Translation With Bilateral Perturbationcited by 34
  20. 2019Emotional speech synthesis with rich and granularized controlcited by 28
  21. 2022GigaST: A 10,000-hour Pseudo Speech Translation Corpuscited by 20
  22. 2022Textless Direct Speech-to-Speech Translation with Discrete Speech Representationcited by 19
  23. 2020Towards Natural Bilingual and Code-Switched Speech Synthesis Based on Mix of Monolingual Recordings and Cross-Lingual Voice Conversioncited by 12
  24. 2019Unsupervised Polyglot Text To Speechcited by 10
  25. 2022Joint Pre-Training with Speech and Bilingual Text for Direct Speech to Speech Translationcited by 9
  26. 2022Cross-Lingual Text-to-Speech Using Multi-Task Learning and Speaker Classifier Joint Trainingcited by 7
  27. 2022Cross-lingual Text-To-Speech with Flow-based Voice Conversion for Improved Pronunciationcited by 3
  28. 2022An Empirical Study on L2 Accents of Cross-lingual Text-to-Speech Systems via Vowel Spacecited by 2

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 27; 2024: 72; 2025: 42; 2026: 26.

167 papers.

Sort:

By area:

  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,433
  2. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 523
  3. 2023Scaling Speech Technology to 1,000+ Languagescited by 402
  4. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 357
  5. 2023Character-LLM: A Trainable Agent for Role-Playingcited by 336
  6. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 335
  7. 2024XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Modelcited by 175
  8. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 144
  9. 2023Audiobox: Unified Audio Generation with Natural Language Promptscited by 120
  10. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 120
  11. 2024FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMscited by 119
  12. 2024VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wildcited by 118
38 more
  1. 2025Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokenscited by 109
  2. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 89
  3. 2023VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translationcited by 87
  4. 2025Automated Movie Generation via Multi-Agent CoT Planningcited by 83
  5. 2024BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of datacited by 75
  6. 2024Natural language guidance of high-fidelity text-to-speech with synthetic annotationscited by 67
  7. 2024Autoregressive Speech Synthesis without Vector Quantizationcited by 65
  8. 2023SpeechX: Neural Codec Language Model as a Versatile Speech Transformercited by 64
  9. 2023Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Biascited by 51
  10. 2024Masked Audio Generation using a Single Non-Autoregressive Transformercited by 50
  11. 2024ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reorderingcited by 50
  12. 2023WavMark: Watermarking for Audio Generationcited by 49
  13. 2024Motion-Agent: A Conversational Framework for Human Motion Generation with LLMscited by 46
  14. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 43
  15. 2025MegaTTS 3: Sparse Alignment Enhanced Latent Diffusion Transformer for Zero-Shot Speech Synthesiscited by 41
  16. 2023Transformers in Speech Processing: A Surveycited by 38
  17. 2024VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignmentcited by 37
  18. 2024Foundation Models for Music: A Surveycited by 35
  19. 2024MovieBench: A Hierarchical Movie Level Dataset for Long Video Generationcited by 35
  20. 2024Autoregressive Diffusion Transformer for Text-to-Speech Synthesiscited by 34
  21. 2024The Codecfake Dataset and Countermeasures for the Universally Detection of Deepfake Audiocited by 31
  22. 2023OpenVoice: Versatile Instant Voice Cloningcited by 30
  23. 2023Speech Translation with Large Language Models: An Industrial Practicecited by 28
  24. 2023Make-A-Voice: Unified Voice Synthesis With Discrete Representationcited by 27
  25. 2025Koel-TTS: Enhancing LLM based Speech Generation with Preference Alignment and Classifier Free Guidancecited by 27
  26. 2025F5R-TTS: Improving Flow-Matching based Text-to-Speech with Group Relative Policy Optimizationcited by 24
  27. 2024FlashSpeech: Efficient Zero-Shot Speech Synthesiscited by 23
  28. 2024SimpleSpeech: Towards Simple and Efficient Text-to-Speech with Scalar Latent Transformer Diffusion Modelscited by 23
  29. 2023SpeechGen: Unlocking the Generative Power of Speech Language Models with Promptscited by 21
  30. 2023Boosting Large Language Model for Speech Synthesis: An Empirical Studycited by 21
  31. 2024Improving Robustness of LLM-based Speech Synthesis by Learning Monotonic Alignmentcited by 21
  32. 2024Preference Alignment Improves Language Model-Based TTScited by 21
  33. 2024Enhancing Zero-shot Text-to-Speech Synthesis with Human Feedbackcited by 20
  34. 2023PolyVoice: Language Models for Speech to Speech Translationcited by 19
  35. 2024SimpleSpeech 2: Towards Simple and Efficient Text-to-Speech with Flow-based Scalar Latent Transformer Diffusion Modelscited by 17
  36. 2024CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversationscited by 15
  37. 2025Differentiable Reward Optimization for LLM based TTS systemcited by 15
  38. 2023AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representationcited by 14

50 most cited of 167 All 167 (JSON)