# Analyzing Acoustic Word Embeddings from Pre-trained Self-supervised Speech Models

arXiv: https://arxiv.org/abs/2210.16043 · HTML: https://mlsanity.com/paper/2210.16043/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 32 references that are in the map.
- First posted on arXiv 28 Oct 2022.

## Builds on (most cited first, 20 of 20)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Libri-Light: A Benchmark for ASR with Limited or No Supervision](https://mlsanity.com/paper/1912.07875/) (2019)
- [Layer-wise Analysis of a Self-supervised Speech Representation Model](https://mlsanity.com/paper/2107.04734/) (2021)
- [Unsupervised Speech Recognition](https://mlsanity.com/paper/2105.11084/) (2021)
- [The Zero Resource Speech Challenge 2017](https://mlsanity.com/paper/1712.04313/) (2017)
- [Towards End-to-end Unsupervised Speech Recognition](https://mlsanity.com/paper/2204.02492/) (2022)
- [An embedded segmental K-means model for unsupervised segmentation and clustering of speech](https://mlsanity.com/paper/1703.08135/) (2017)
- [Truly unsupervised acoustic word embeddings using weak top-down constraints in encoder-decoder models](https://mlsanity.com/paper/1811.00403/) (2018)
- [Deep convolutional acoustic word embeddings using word-pair side information](https://mlsanity.com/paper/1510.01032/) (2015)
- [A segmental framework for fully-unsupervised large-vocabulary speech recognition](https://mlsanity.com/paper/1606.06950/) (2016)
- [Discriminative Acoustic Word Embeddings: Recurrent Neural Network-Based Approaches](https://mlsanity.com/paper/1611.02550/) (2016)
- [Query-by-Example Search with Discriminative Neural Acoustic Word Embeddings](https://mlsanity.com/paper/1706.03818/) (2017)
- [Acoustic word embeddings for zero-resource languages using self-supervised contrastive learning and multilingual adaptation](https://mlsanity.com/paper/2103.10731/) (2021)
- [Multilingual Jointly Trained Acoustic and Written Word Embeddings](https://mlsanity.com/paper/2006.14007/) (2020)
- [A Correspondence Variational Autoencoder for Unsupervised Acoustic Word Embeddings](https://mlsanity.com/paper/2012.02221/) (2020)
- [A comparison of self-supervised speech representations as input features for unsupervised acoustic word embeddings](https://mlsanity.com/paper/2012.07387/) (2020)
- [Speech Sequence Embeddings using Nearest Neighbors Contrastive Learning](https://mlsanity.com/paper/2204.05148/) (2022)
- [Unsupervised neural and Bayesian models for zero-resource speech processing](https://mlsanity.com/paper/1701.00851/) (2017)

## Built on by (most cited first, 21 of 21)

- [Comparative layer-wise analysis of self-supervised speech models](https://mlsanity.com/paper/2211.03929/) (2022)
- [Self-Supervised Speech Representations are More Phonetic than Semantic](https://mlsanity.com/paper/2406.08619/) (2024)
- [What Do Self-Supervised Speech Models Know About Words?](https://mlsanity.com/paper/2307.00162/) (2023)
- [Self-supervised Predictive Coding Models Encode Speaker and Phonetic Information in Orthogonal Subspaces](https://mlsanity.com/paper/2305.12464/) (2023)
- [Emergent morpho-phonological representations in self-supervised speech models](https://mlsanity.com/paper/2509.22973/) (2025)
- [Crossmodal ASR Error Correction with Discrete Speech Units](https://mlsanity.com/paper/2405.16677/) (2024)
- [Layer-Wise Analysis of Self-Supervised Acoustic Word Embeddings: A Study on Speech Emotion Recognition](https://mlsanity.com/paper/2402.02617/) (2024)
- [What Do Speech Foundation Models Not Learn About Speech?](https://mlsanity.com/paper/2410.12948/) (2024)
- [Unsupervised Word Discovery: Boundary Detection with Clustering vs. Dynamic Programming](https://mlsanity.com/paper/2409.14486/) (2024)
- [Segmentation-Variant Codebooks for Preservation of Paralinguistic and Prosodic Information](https://mlsanity.com/paper/2505.15667/) (2025)
- [Should Top-Down Clustering Affect Boundaries in Unsupervised Word Discovery?](https://mlsanity.com/paper/2507.19204/) (2025)
- [Iterative refinement, not training objective, makes HuBERT behave differently from wav2vec 2.0](https://mlsanity.com/paper/2508.08110/) (2025)
- [Unsupervised lexicon learning from speech is limited by representations rather than clustering](https://mlsanity.com/paper/2510.09225/) (2025)
- [Revisiting speech segmentation and lexicon learning with better features](https://mlsanity.com/paper/2401.17902/) (2024)
- [Improving Acoustic Word Embeddings through Correspondence Training of Self-supervised Speech Representations](https://mlsanity.com/paper/2403.08738/) (2024)
- [Discrete Speech Unit Extraction via Independent Component Analysis](https://mlsanity.com/paper/2501.06562/) (2025)
- [Acoustic Word Embeddings for Untranscribed Target Languages with Continued Pretraining and Learned Pooling](https://mlsanity.com/paper/2306.02153/) (2023)
- [Leveraging multilingual transfer for unsupervised semantic acoustic word embeddings](https://mlsanity.com/paper/2307.02083/) (2023)
- [Enhancing Few-shot Keyword Spotting Performance through Pre-Trained Self-supervised Speech Models](https://mlsanity.com/paper/2506.17686/) (2025)
- [Do Self-Supervised Speech Models Exhibit the Critical Period Effects in Language Acquisition?](https://mlsanity.com/paper/2508.21210/) (2025)
- [Learning task-specific subspaces via interventional post-training of speech foundation models](https://mlsanity.com/paper/2606.17967/) (2026)

## Abstract (arXiv)

Given the strong results of self-supervised models on various tasks, there have been surprisingly few studies exploring self-supervised representations for acoustic word embeddings (AWE), fixed-dimensional vectors representing variable-length spoken word segments. In this work, we study several pre-trained models and pooling methods for constructing AWEs with self-supervised representations. Owing to the contextualized nature of self-supervised representations, we hypothesize that simple pooling methods, such as averaging, might already be useful for constructing AWEs. When evaluating on a standard word discrimination task, we find that HuBERT representations with mean-pooling rival the state of the art on English AWEs. More surprisingly, despite being trained only on English, HuBERT representations evaluated on Xitsonga, Mandarin, and French consistently outperform the multilingual model XLSR-53 (as well as Wav2Vec 2.0 trained on English).