# Improving Metrics for Speech Translation

arXiv: https://arxiv.org/abs/2305.12918 · HTML: https://mlsanity.com/paper/2305.12918/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 21 references that are in the map.
- First posted on arXiv 22 May 2023.

## Builds on (most cited first, 9 of 9)

- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [fairseq S2T: Fast Speech-to-Text Modeling with fairseq](https://mlsanity.com/paper/2010.05171/) (2020)
- [Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing](https://mlsanity.com/paper/2004.14564/) (2020)
- [BLEU might be Guilty but References are not Innocent](https://mlsanity.com/paper/2004.06063/) (2020)
- [Paraphrase Generation as Zero-Shot Multilingual Translation: Disentangling Semantic Similarity from Lexical and Syntactic Diversity](https://mlsanity.com/paper/2008.04935/) (2020)
- [SDS-200: A Swiss German Speech to Standard German Text Corpus](https://mlsanity.com/paper/2205.09501/) (2022)
- [Swiss Parliaments Corpus, an Automatically Aligned Swiss German Speech to Standard German Text Corpus](https://mlsanity.com/paper/2010.02810/) (2020)
- [Generative Pretraining for Paraphrase Evaluation](https://mlsanity.com/paper/2107.08251/) (2021)

## Built on by (most cited first, 1 of 1)

- [Dialect Transfer for Swiss German Speech Translation](https://mlsanity.com/paper/2310.09088/) (2023)

## Abstract (arXiv)

We introduce Parallel Paraphrasing (Paraboth), an augmentation method for translation metrics making use of automatic paraphrasing of both the reference and hypothesis. This method counteracts the typically misleading results of speech translation metrics such as WER, CER, and BLEU if only a single reference is available. We introduce two new datasets explicitly created to measure the quality of metrics intended to be applied to Swiss German speech-to-text systems. Based on these datasets, we show that we are able to significantly improve the correlation with human quality perception if our method is applied to commonly used metrics.