Improving Metrics for Speech Translation
Not on the map. Why?
Abstract (arXiv)
We introduce Parallel Paraphrasing (Paraboth), an augmentation method for translation metrics making use of automatic paraphrasing of both the reference and hypothesis. This method counteracts the typically misleading results of speech translation metrics such as WER, CER, and BLEU if only a single reference is available. We introduce two new datasets explicitly created to measure the quality of metrics intended to be applied to Swiss German speech-to-text systems. Based on these datasets, we show that we are able to significantly improve the correlation with human quality perception if our method is applied to commonly used metrics.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 9 of its 21 references that are in the map. First posted on arXiv 22 May 2023.
Builds on
9 of 21 references.
References · 9
- 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
- 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
- 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 176
- 2020Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasingcited by 115
- 2020BLEU might be Guilty but References are not Innocentcited by 110
- 2020Paraphrase Generation as Zero-Shot Multilingual Translation: Disentangling Semantic Similarity from Lexical and Syntactic Diversitycited by 35
- 2022SDS-200: A Swiss German Speech to Standard German Text Corpuscited by 22
- 2020Swiss Parliaments Corpus, an Automatically Aligned Swiss German Speech to Standard German Text Corpuscited by 16
- 2021Generative Pretraining for Paraphrase Evaluationcited by 4
Built on by
Citing papers per year: 2023: 1.
1 paper.
- 2023Dialect Transfer for Swiss German Speech Translationcited by 7
- 2023Dialect Transfer for Swiss German Speech Translation13 Oct 2023