mlsanityData to 7 Oct 2026

Benchmarking Speech-to-Speech Translation Models

arXiv 2606.03241 ↗First posted 2 Jun 2026cs.CL eess.AScited by 0

Not on the map. Why?

Abstract (arXiv)

Speech-to-speech translation (S2ST) has advanced rapidly, but offline evaluation lacks a unified protocol: studies report non-overlapping metric subsets, preventing direct comparisons. We introduce COMPASS, a unified and reproducible benchmarking framework integrating 46 metrics across eight dimensions, and deploy it on 1,248 model-language configurations from FLEURS and CVSS, spanning cascaded and end-to-end architectures over ten language pairs. Architectures exhibit complementary strengths: best-vs-worst gaps exceed 30% on naturalness and speaker preservation but remain within a few points on translation quality, so single-metric rankings systematically misrepresent system quality. Correlation filtering reduces 46 metrics to 10 per direction, with three axes requiring different metrics across X→EN and EN→X (e.g., TER/UTMOS vs. ChrF++/NISQA-MOS); these subsets preserve rankings (Spearman's ρ>0.80) while cutting evaluation time by ≈ 2.5×. Human validation across dubbing, podcasts, and medical domains shows standalone MOS predictors fail to predict listener preference, while top domain-specific metrics correlate with human judgment (ρ ≥ 0.90). We release COMPASS as a foundation for domain-aware S2ST evaluation.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 37 of its 77 references that are in the map. First posted on arXiv 2 Jun 2026.

Builds on

37 of 77 references.

References · 37

  1. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  6. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  7. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  8. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  9. 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 606
  10. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  11. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
  12. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 381
25 more
  1. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 333
  2. 2023emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representationcited by 196
  3. 2021NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasetscited by 189
  4. 2020Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metricscited by 155
  5. 2021Direct speech-to-speech translation with discrete unitscited by 148
  6. 2025CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-trainingcited by 142
  7. 2019Direct speech-to-speech translation with a sequence-to-sequence modelcited by 118
  8. 2020SimulEval: An Evaluation Toolkit for Simultaneous Translationcited by 94
  9. 2021Translatotron 2: High-quality direct speech-to-speech translation with voice preservationcited by 69
  10. 2021Generalization Ability of MOS Prediction Networkscited by 69
  11. 2022Over-Generation Cannot Be Rewarded: Length-Adaptive Average Lagging for Simultaneous Speech Translationcited by 56
  12. 2021Cascade versus Direct Speech Translation: Do the Differences Still Make a Difference?cited by 54
  13. 2021Isometric MT: Neural Machine Translation for Automatic Dubbingcited by 18
  14. 2022Dubbing in Practice: A Large Scale Study of Human Localization With Insights for Automatic Dubbingcited by 17
  15. 2022BLASER: A Text-Free Speech-to-Speech Translation Evaluation Metriccited by 16
  16. 2023Translatotron 3: Speech to Speech Translation with Monolingual Datacited by 12
  17. 2024TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservationcited by 10
  18. 2025Aligning ASR Evaluation with Human and LLM Judgments: Intelligibility Metrics Using Phonetic, Semantic, and NLI Approachescited by 7
  19. 2025UniSS: Unified Expressive Speech-to-Speech Translation with Your Voicecited by 6
  20. 2023Jointly Optimizing Translations and Speech Timing to Improve Isochrony in Automatic Dubbingcited by 5
  21. 2024MELD-ST: An Emotion-aware Speech Translation Datasetcited by 4
  22. 2025Simulstream: Open-Source Toolkit for Evaluation and Demonstration of Streaming Speech-to-Text Translation Systemscited by 4
  23. 2025MultiMed-ST: Large-scale Many-to-many Multilingual Medical Speech Translationcited by 2
  24. 2025SLM-S2ST: A multimodal language model for direct speech-to-speech translationcited by 2
  25. 2024IsoChronoMeter: A simple and effective isochronic translation evaluation metriccited by 1

Built on by

Not cited yet.