mlsanityData to 14 Oct 2026

Cross-Lingual F5-TTS: Towards Language-Agnostic Voice Cloning and Speech Synthesis

arXiv 2509.14579 ↗First posted 18 Sep 2025cs.SDcited by 5

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Flow-matching-based text-to-speech (TTS) models have shown high-quality speech synthesis. However, most current flow-matching-based TTS models still rely on reference transcripts corresponding to the audio prompt for synthesis. This dependency prevents cross-lingual voice cloning when audio prompt transcripts are unavailable, particularly for unseen languages. The key challenges for flow-matching-based TTS models to remove audio prompt transcripts are identifying word boundaries during training and determining appropriate duration during inference. In this paper, we introduce Cross-Lingual F5-TTS, a framework that enables cross-lingual voice cloning without audio prompt transcripts. Our method preprocesses audio prompts by forced alignment to obtain word boundaries, enabling direct synthesis from audio prompts while excluding transcripts during training. To address the duration modeling challenge, we train speaking rate predictors at different linguistic granularities to derive duration from speaker pace. Experiments show that our approach matches the performance of F5-TTS while enabling cross-lingual voice cloning.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 23 of its 26 references that are in the map. First posted on arXiv 18 Sep 2025.

Builds on

23 of 26 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generation6 Feb
  2. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models13 Dec
  3. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching9 Oct
  4. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models5 Mar
  5. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers18 Apr

Other references · 18

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  2. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  3. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  4. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
  5. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 469
  6. 2023Scaling Speech Technology to 1,000+ Languagescited by 411
  7. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 390
  8. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 379
  9. 2023WhisperX: Time-Accurate Speech Transcription of Long-Form Audiocited by 326
  10. 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 300
  11. 2024Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generationcited by 204
  12. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 167
6 more
  1. 2023Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesiscited by 153
  2. 2024E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTScited by 152
  3. 2022Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognitioncited by 144
  4. 2024Autoregressive Speech Synthesis without Vector Quantizationcited by 65
  5. 2023LibriSpeech-PC: Benchmark for Evaluation of Punctuation and Capitalization Capabilities of end-to-end ASR Modelscited by 49
  6. 2025MiniMax-Speech: Intrinsic Zero-Shot Text-to-Speech with a Learnable Speaker Encodercited by 48

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026X-Voice: Enabling Everyone to Speak 30 Languages via Zero-Shot Cross-Lingual Voice Cloningcited by 4
  2. 2026VoXtream2: Full-stream TTS with dynamic speaking rate controlcited by 3
  3. 2026Joint Residual Reweighting for Classifier Free Guidance in Flow-Matching Zero-Shot TTScited by 1
  4. 2026Confucius4-TTS: Transcript-Free Cross-Lingual Zero-Shot TTS with a Learnable Speaker Encodercited by 0
  5. 2026Cross-Lingual TTS for Indian Languages: A Benchmark and Inference-Time Duration Estimation Strategiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026PFluxTTS: Hybrid Flow-Matching TTS with Robust Cross-Lingual Voice Cloning and Inference-Time Model Fusion
  2. 2026KIT's Submission to Cross-Lingual Voice Cloning in IWSLT 2026
  3. 2026One Voice, Many Tongues: Cross-Lingual Voice Cloning for Scientific Speech
  4. 2026Tacit-TTS: From Autoregressive Decoding to Masked Prediction for Efficient Transcript-Free Voice Cloning
  5. 2023Cross-lingual Knowledge Distillation via Flow-based Voice Conversion for Robust Polyglot Text-To-Speech
  6. 2026Adding Voice Cloning to Text-to-Audio-Video Models with a Single Zero-Initialised Layer
  7. 2025LatinX: Aligning a Multilingual TTS Model with Direct Preference Optimization
  8. 2026Training a Compact, Strong, and Realistic Text-to-Speech Model