# Assessing Evaluation Metrics for Speech-to-Speech Translation

arXiv: https://arxiv.org/abs/2110.13877 · HTML: https://mlsanity.com/paper/2110.13877/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 37 references that are in the map.
- First posted on arXiv 26 Oct 2021.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [Six Challenges for Neural Machine Translation](https://mlsanity.com/paper/1706.03872/) (2017)
- [Direct speech-to-speech translation with a sequence-to-sequence model](https://mlsanity.com/paper/1904.06037/) (2019)
- [Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder](https://mlsanity.com/paper/1611.04798/) (2016)
- [Speech Recognition with Augmented Synthesized Speech](https://mlsanity.com/paper/1909.11699/) (2019)
- [CSS10: A Collection of Single Speaker Speech Datasets for 10 Languages](https://mlsanity.com/paper/1903.11269/) (2019)
- [UWSpeech: Speech to Speech Translation for Unwritten Languages](https://mlsanity.com/paper/2006.07926/) (2020)
- [Wave-Tacotron: Spectrogram-free end-to-end text-to-speech synthesis](https://mlsanity.com/paper/2011.03568/) (2020)
- [SwissDial: Parallel Multidialectal Corpus of Spoken Swiss German](https://mlsanity.com/paper/2103.11401/) (2021)
- [A Swiss German Dictionary: Variation in Speech and Writing](https://mlsanity.com/paper/2004.00139/) (2020)

## Built on by (most cited first, 10 of 10)

- [UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Units](https://mlsanity.com/paper/2212.08055/) (2022)
- [Towards Building Text-To-Speech Systems for the Next Billion Users](https://mlsanity.com/paper/2211.09536/) (2022)
- [BLASER: A Text-Free Speech-to-Speech Translation Evaluation Metric](https://mlsanity.com/paper/2212.08486/) (2022)
- [A Textless Metric for Speech-to-Speech Comparison](https://mlsanity.com/paper/2210.11835/) (2022)
- [A Collection of Pragmatic-Similarity Judgments over Spoken Dialog Utterances](https://mlsanity.com/paper/2403.14808/) (2024)
- [Kahaani: A Multimodal Co-Creative Storytelling System](https://mlsanity.com/paper/2409.11261/) (2024)
- [OpenSTBench: Beyond Semantic Evaluation for Speech Translation](https://mlsanity.com/paper/2605.30792/) (2026)
- [Applying Automated Machine Translation to Educational Video Courses](https://mlsanity.com/paper/2301.03141/) (2023)
- [Audio-Based Crowd-Sourced Evaluation of Machine Translation Quality](https://mlsanity.com/paper/2509.14023/) (2025)
- [ASTRA: A Scalable Next-Generation ATCO Training Simulator with Autonomous Simpilots](https://mlsanity.com/paper/2606.18319/) (2026)

## Abstract (arXiv)

Speech-to-speech translation combines machine translation with speech synthesis, introducing evaluation challenges not present in either task alone. How to automatically evaluate speech-to-speech translation is an open question which has not previously been explored. Translating to speech rather than to text is often motivated by unwritten languages or languages without standardized orthographies. However, we show that the previously used automatic metric for this task is best equipped for standardized high-resource languages only. In this work, we first evaluate current metrics for speech-to-speech translation, and second assess how translation to dialectal variants rather than to standardized languages impacts various evaluation methods.