mlsanityData to 7 Oct 2026

A Textless Metric for Speech-to-Speech Comparison

arXiv 2210.11835 ↗First posted 21 Oct 2022cs.CL cs.SD eess.AScited by 4

Not on the map. Why?

Abstract (arXiv)

In this paper, we introduce a new and simple method for comparing speech utterances without relying on text transcripts. Our speech-to-speech comparison metric utilizes state-of-the-art speech2unit encoders like HuBERT to convert speech utterances into discrete acoustic units. We then propose a simple and easily replicable neural architecture that learns a speech-based metric that closely corresponds to its text-based counterpart. This textless metric has numerous potential applications, including evaluating speech-to-speech translation for oral languages, languages without dependable ASR systems, or to avoid the need for ASR transcription altogether. This paper also shows that for speech-to-speech translation evaluation, ASR-BLEU (which consists in automatically transcribing both speech hypothesis and reference and compute sentence-level BLEU between transcripts) is a poor proxy to real text-BLEU even when ASR system is strong.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 15 of its 23 references that are in the map. First posted on arXiv 21 Oct 2022.

Builds on

15 of 23 references.

References · 15

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  6. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  7. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  8. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  9. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 404
  10. 2021Direct speech-to-speech translation with discrete unitscited by 148
  11. 2019Direct speech-to-speech translation with a sequence-to-sequence modelcited by 118
  12. 2022CVSS Corpus and Massively Multilingual Speech-to-Speech Translationcited by 71
3 more
  1. 2022textless-lib: a Library for Textless Spoken Language Processingcited by 28
  2. 2022Speech Resources in the Tamasheq Languagecited by 13
  3. 2021Assessing Evaluation Metrics for Speech-to-Speech Translationcited by 10

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 1; 2024: 2.

4 papers.

  1. 2022BLASER: A Text-Free Speech-to-Speech Translation Evaluation Metriccited by 16
  2. 2023Textless Speech-to-Speech Translation With Limited Parallel Datacited by 3
  3. 2024A Collection of Pragmatic-Similarity Judgments over Spoken Dialog Utterancescited by 2
  4. 2024Towards Precision Characterization of Communication Disorders using Models of Perceived Pragmatic Similaritycited by 0