# Improving Acoustic Word Embeddings through Correspondence Training of Self-supervised Speech Representations

arXiv: https://arxiv.org/abs/2403.08738 · HTML: https://mlsanity.com/paper/2403.08738/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 28 references that are in the map.
- First posted on arXiv 13 Mar 2024.

## Builds on (most cited first, 21 of 21)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [MLS: A Large-Scale Multilingual Dataset for Speech Research](https://mlsanity.com/paper/2012.03411/) (2020)
- [Truly unsupervised acoustic word embeddings using weak top-down constraints in encoder-decoder models](https://mlsanity.com/paper/1811.00403/) (2018)
- [Query-by-Example Search with Discriminative Neural Acoustic Word Embeddings](https://mlsanity.com/paper/1706.03818/) (2017)
- [Analyzing Acoustic Word Embeddings from Pre-trained Self-supervised Speech Models](https://mlsanity.com/paper/2210.16043/) (2022)
- [Multi-view Recurrent Neural Acoustic Word Embeddings](https://mlsanity.com/paper/1611.04496/) (2016)
- [Learning Audio-Text Agreement for Open-vocabulary Keyword Spotting](https://mlsanity.com/paper/2206.15400/) (2022)
- [Improved acoustic word embeddings for zero-resource languages using multilingual transfer](https://mlsanity.com/paper/2006.02295/) (2020)
- [Do Acoustic Word Embeddings Capture Phonological Similarity? An Empirical Study](https://mlsanity.com/paper/2106.08686/) (2021)
- [How Familiar Does That Sound? Cross-Lingual Representational Similarity Analysis of Acoustic Word Embeddings](https://mlsanity.com/paper/2109.10179/) (2021)
- [Learning Acoustic Word Embeddings with Temporal Context for Query-by-Example Speech Search](https://mlsanity.com/paper/1806.03621/) (2018)
- [Analyzing autoencoder-based acoustic word embeddings](https://mlsanity.com/paper/2004.01647/) (2020)
- [Acoustic span embeddings for multilingual query-by-example search](https://mlsanity.com/paper/2011.11807/) (2020)
- [Towards hate speech detection in low-resource languages: Comparing ASR to acoustic word embeddings on Wolof and Swahili](https://mlsanity.com/paper/2306.00410/) (2023)
- [SCORE: Self-supervised Correspondence Fine-tuning for Improved Content Representations](https://mlsanity.com/paper/2403.06260/) (2024)
- [Integrating Form and Meaning: A Multi-Task Learning Model for Acoustic Word Embeddings](https://mlsanity.com/paper/2209.06633/) (2022)
- [Self-Supervised Acoustic Word Embedding Learning via Correspondence Transformer Encoder](https://mlsanity.com/paper/2307.09871/) (2023)

## Built on by (most cited first, 3 of 3)

- [LASER: Learning by Aligning Self-supervised Representations of Speech for Improving Content-related Tasks](https://mlsanity.com/paper/2406.09153/) (2024)
- [Phoneme- and Word-Level Metrics Using Self-Supervised Speech Representations for Forced Alignment Evaluation](https://mlsanity.com/paper/2608.28508/) (2026)
- [Position-invariant Fine-tuning of Speech Enhancement Models with Self-supervised Speech Representations](https://mlsanity.com/paper/2601.21084/) (2026)

## Abstract (arXiv)

Acoustic word embeddings (AWEs) are vector representations of spoken words. An effective method for obtaining AWEs is the Correspondence Auto-Encoder (CAE). In the past, the CAE method has been associated with traditional MFCC features. Representations obtained from self-supervised learning (SSL)-based speech models such as HuBERT, Wav2vec2, etc., are outperforming MFCC in many downstream tasks. However, they have not been well studied in the context of learning AWEs. This work explores the effectiveness of CAE with SSL-based speech representations to obtain improved AWEs. Additionally, the capabilities of SSL-based speech models are explored in cross-lingual scenarios for obtaining AWEs. Experiments are conducted on five languages: Polish, Portuguese, Spanish, French, and English. HuBERT-based CAE model achieves the best results for word discrimination in all languages, despite Hu-BERT being pre-trained on English only. Also, the HuBERT-based CAE model works well in cross-lingual settings. It outperforms MFCC-based CAE models trained on the target languages when trained on one source language and tested on target languages.