mlsanityData to 7 Oct 2026

X-Translator: A Real-Time Multilingual Speaker-Aware Speech-to-Speech Translation System

arXiv 2607.17544 ↗First posted 20 Jul 2026eess.AS cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Real-time speech-to-speech translation (S2ST) systems must balance translation quality, latency, speech naturalness, and speaker consistency. Publicly documented S2ST systems have advanced direct, multilingual, streaming, and expressive modeling, while proprietary products and APIs increasingly expose real-time translation capabilities to users. However, practical deployment remains challenging for open and reproducible systems, especially in long-form and multi-speaker conversations where partial ASR hypotheses are unstable, turn boundaries are ambiguous, and target speech must be generated with an appropriate speaker prompt. We present X-Translator, a low-cost modular cascaded S2ST system that combines streaming ASR, machine translation, and prompt-conditioned TTS through a session-level runtime controller. The system uses incremental segment commitment to convert unstable ASR streams into translation-ready units, and an online speaker prompt manager to bind source speech spans to speaker-specific voice prompts for synthesis. We evaluate translation, speech quality, and latency with OpenSTBench, compare against proprietary speech translation APIs as behavioral baselines, measure long-form voice stability, evaluate speaker preservation in multi-speaker conversations, and assess multilingual translation quality. X-Translator provides an open platform for understanding the practical trade-offs of deployment-oriented S2ST. Code and demo are available at https://github.com/zhaoyx239/X-Translator.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 25 of its 33 references that are in the map. First posted on arXiv 20 Jul 2026.

Builds on

25 of 33 references.

References · 25

  1. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
  2. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  3. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 353
  4. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 323
  5. 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 224
  6. 2021Direct speech-to-speech translation with discrete unitscited by 148
  7. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  8. 2019Direct speech-to-speech translation with a sequence-to-sequence modelcited by 118
  9. 2023CAM++: A Fast and Efficient Network for Speaker Verification Using Context-Aware Maskingcited by 95
  10. 2020SimulEval: An Evaluation Toolkit for Simultaneous Translationcited by 94
  11. 2019Monotonic Multihead Attentioncited by 91
  12. 2025IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speechcited by 85
13 more
  1. 2020Spot the conversation: speaker diarisation in the wildcited by 75
  2. 2021Translatotron 2: High-quality direct speech-to-speech translation with voice preservationcited by 69
  3. 2020Low-Latency Sequence-to-Sequence Speech Recognition and Translation by Partial Hypothesis Selectioncited by 47
  4. 2022SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translationscited by 35
  5. 2023AlignAtt: Using Attention-based Audio-Translation Alignments as a Guide for Simultaneous Speech Translationcited by 35
  6. 2022Attention as a Guide for Simultaneous Speech Translationcited by 32
  7. 2021Direct Simultaneous Speech-to-Speech Translation with Variational Monotonic Multihead Attentioncited by 9
  8. 2023DASpeech: Directed Acyclic Transformer for Fast and High-quality Speech-to-Speech Translationcited by 9
  9. 2025Seed LiveInterpret 2.0: End-to-end Simultaneous Speech-to-speech Translation with Your Voicecited by 9
  10. 2025UniSS: Unified Expressive Speech-to-Speech Translation with Your Voicecited by 6
  11. 2025NiuTrans.LMT: Toward Inclusive and Scalable Multilingual Machine Translation with LLMscited by 4
  12. 2026X-Voice: Enabling Everyone to Speak 30 Languages via Zero-Shot Cross-Lingual Voice Cloningcited by 4
  13. 2026OpenSTBench: Beyond Semantic Evaluation for Speech Translationcited by 1

Built on by

Not cited yet.