mlsanityData to 7 Oct 2026

Evaluating the reliability of acoustic speech embeddings

arXiv 2007.13542 ↗First posted 27 Jul 2020eess.AS cs.CL cs.LGcited by 21

Not on the map. Why?

Abstract (arXiv)

Speech embeddings are fixed-size acoustic representations of variable-length speech sequences. They are increasingly used for a variety of tasks ranging from information retrieval to unsupervised term discovery and speech segmentation. However, there is currently no clear methodology to compare or optimise the quality of these embeddings in a task-neutral way. Here, we systematically compare two popular metrics, ABX discrimination and Mean Average Precision (MAP), on 5 languages across 17 embedding methods, ranging from supervised to fully unsupervised, and using different loss functions (autoencoders, correspondence autoencoders, siamese). Then we use the ABX and MAP to predict performances on a new downstream task: the unsupervised estimation of the frequencies of speech segments in a given corpus. We find that overall, ABX and MAP correlate with one another and with frequency estimation. However, substantial discrepancies appear in the fine-grained distinctions across languages and/or embedding methods. This makes it unrealistic at present to propose a task-independent silver bullet method for computing the intrinsic quality of speech embeddings. There is a need for more detailed analysis of the metrics currently used to evaluate such embeddings.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). First posted on arXiv 27 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 6; 2022: 6; 2023: 8; 2024: 0; 2025: 1.

21 papers.

Sort:

  1. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
  2. 2023What Do Self-Supervised Speech Models Know About Words?cited by 62
  3. 2022Self-supervised language learning from raw audio: Lessons from the Zero Resource Speech Challengecited by 34
  4. 2023Speech Self-Supervised Representation Benchmarking: Are We Doing it Right?cited by 28
  5. 2021Acoustic word embeddings for zero-resource languages using self-supervised contrastive learning and multilingual adaptationcited by 19
  6. 2023Speech Self-Supervised Representations Benchmarking: a Case for Larger Probing Headscited by 16
  7. 2023Fine-tuning Strategies for Faster Inference using Speech Self-Supervised Models: A Comparative Studycited by 15
  8. 2022DP-Parse: Finding Word Boundaries from Raw Speech with an Instance Lexiconcited by 12
  9. 2022Evaluating context-invariance in unsupervised speech representationscited by 11
  10. 2021Multilingual transfer of acoustic word embeddings improves when training on languages related to the target zero-resource languagecited by 7
  11. 2021Pretext Tasks selection for multitask self-supervised speech representation learningcited by 7
  12. 2022Speech Sequence Embeddings using Nearest Neighbors Contrastive Learningcited by 7
9 more
  1. 2025SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervisioncited by 6
  2. 2022Automatic Data Augmentation Selection and Parametrization in Contrastive Self-Supervised Speech Representation Learningcited by 4
  3. 2021Conditional independence for pretext task selection in Self-supervised speech representation learningcited by 3
  4. 2021Discrete representations in neural models of spoken languagecited by 3
  5. 2022Integrating Form and Meaning: A Multi-Task Learning Model for Acoustic Word Embeddingscited by 2
  6. 2023Analyzing the Representational Geometry of Acoustic Word Embeddingscited by 2
  7. 2023Phone and speaker spatial organization in self-supervised speech representationscited by 2
  8. 2023Acoustic Word Embeddings for Untranscribed Target Languages with Continued Pretraining and Learned Poolingcited by 2
  9. 2023Automatic Data Augmentation for Domain Adapted Fine-Tuning of Self-Supervised Speech Representationscited by 0