# Leveraging multilingual transfer for unsupervised semantic acoustic word embeddings

arXiv: https://arxiv.org/abs/2307.02083 · HTML: https://mlsanity.com/paper/2307.02083/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 39 references that are in the map.
- First posted on arXiv 5 Jul 2023.

## Builds on (most cited first, 30 of 30)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)
- [Word Translation Without Parallel Data](https://mlsanity.com/paper/1710.04087/) (2017)
- [Exploiting Similarities among Languages for Machine Translation](https://mlsanity.com/paper/1309.4168/) (2013)
- [SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimation](https://mlsanity.com/paper/1408.3456/) (2014)
- [Deep Multimodal Semantic Embeddings for Speech and Images](https://mlsanity.com/paper/1511.03690/) (2015)
- [Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speech](https://mlsanity.com/paper/1803.08976/) (2018)
- [An embedded segmental K-means model for unsupervised segmentation and clustering of speech](https://mlsanity.com/paper/1703.08135/) (2017)
- [Truly unsupervised acoustic word embeddings using weak top-down constraints in encoder-decoder models](https://mlsanity.com/paper/1811.00403/) (2018)
- [Deep convolutional acoustic word embeddings using word-pair side information](https://mlsanity.com/paper/1510.01032/) (2015)
- [Discriminative Acoustic Word Embeddings: Recurrent Neural Network-Based Approaches](https://mlsanity.com/paper/1611.02550/) (2016)
- [Word Segmentation on Discovered Phone Units with Dynamic Programming and Self-Supervised Scoring](https://mlsanity.com/paper/2202.11929/) (2022)
- [Query-by-Example Search with Discriminative Neural Acoustic Word Embeddings](https://mlsanity.com/paper/1706.03818/) (2017)
- [Analyzing Acoustic Word Embeddings from Pre-trained Self-supervised Speech Models](https://mlsanity.com/paper/2210.16043/) (2022)
- [Semantic speech retrieval with a visually grounded model of untranscribed speech](https://mlsanity.com/paper/1710.01949/) (2017)
- [Acoustic word embeddings for zero-resource languages using self-supervised contrastive learning and multilingual adaptation](https://mlsanity.com/paper/2103.10731/) (2021)
- [Contrastive prediction strategies for unsupervised segmentation and categorization of phonemes and words](https://mlsanity.com/paper/2110.15909/) (2021)
- [Visually grounded learning of keyword prediction from untranscribed speech](https://mlsanity.com/paper/1703.08136/) (2017)
- [Multilingual Jointly Trained Acoustic and Written Word Embeddings](https://mlsanity.com/paper/2006.14007/) (2020)
- [Phonetic-and-Semantic Embedding of Spoken Words with Applications in Spoken Content Retrieval](https://mlsanity.com/paper/1807.08089/) (2018)
- [Improved acoustic word embeddings for zero-resource languages using multilingual transfer](https://mlsanity.com/paper/2006.02295/) (2020)
- [A comparison of self-supervised speech representations as input features for unsupervised acoustic word embeddings](https://mlsanity.com/paper/2012.07387/) (2020)
- [Multilingual acoustic word embedding models for processing zero-resource languages](https://mlsanity.com/paper/2002.02109/) (2020)
- [Multilingual transfer of acoustic word embeddings improves when training on languages related to the target zero-resource language](https://mlsanity.com/paper/2106.12834/) (2021)
- [Unsupervised Word Segmentation using K Nearest Neighbors](https://mlsanity.com/paper/2204.13094/) (2022)
- [Towards hate speech detection in low-resource languages: Comparing ASR to acoustic word embeddings on Wolof and Swahili](https://mlsanity.com/paper/2306.00410/) (2023)
- [Integrating Form and Meaning: A Multi-Task Learning Model for Acoustic Word Embeddings](https://mlsanity.com/paper/2209.06633/) (2022)

## Built on by (most cited first, 2 of 2)

- [Unsupervised lexicon learning from speech is limited by representations rather than clustering](https://mlsanity.com/paper/2510.09225/) (2025)
- [Recovering the Zipfian Distribution in Unsupervised Term Discovery](https://mlsanity.com/paper/2606.10781/) (2026)

## Abstract (arXiv)

Acoustic word embeddings (AWEs) are fixed-dimensional vector representations of speech segments that encode phonetic content so that different realisations of the same word have similar embeddings. In this paper we explore semantic AWE modelling. These AWEs should not only capture phonetics but also the meaning of a word (similar to textual word embeddings). We consider the scenario where we only have untranscribed speech in a target language. We introduce a number of strategies leveraging a pre-trained multilingual AWE model -- a phonetic AWE model trained on labelled data from multiple languages excluding the target. Our best semantic AWE approach involves clustering word segments using the multilingual AWE model, deriving soft pseudo-word labels from the cluster centroids, and then training a Skipgram-like model on the soft vectors. In an intrinsic word similarity task measuring semantics, this multilingual transfer approach outperforms all previous semantic AWE methods. We also show -- for the first time -- that AWEs can be used for downstream semantic query-by-example search.