# Acoustic word embeddings for zero-resource languages using self-supervised contrastive learning and multilingual adaptation

arXiv: https://arxiv.org/abs/2103.10731 · HTML: https://mlsanity.com/paper/2103.10731/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 61 references that are in the map.
- First posted on arXiv 19 Mar 2021.

## Builds on (most cited first, 30 of 42)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [In Defense of the Triplet Loss for Person Re-Identification](https://mlsanity.com/paper/1703.07737/) (2017)
- [vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations](https://mlsanity.com/paper/1910.05453/) (2019)
- [Multi-task Self-Supervised Visual Learning](https://mlsanity.com/paper/1708.07860/) (2017)
- [Multi-task self-supervised learning for Robust Speech Recognition](https://mlsanity.com/paper/2001.09239/) (2020)
- [End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures](https://mlsanity.com/paper/1911.08460/) (2019)
- [Learning Problem-agnostic Speech Representations from Multiple Self-supervised Tasks](https://mlsanity.com/paper/1904.03416/) (2019)
- [A critical analysis of self-supervision, or what we can learn from a single image](https://mlsanity.com/paper/1904.13132/) (2019)
- [Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speech](https://mlsanity.com/paper/1803.08976/) (2018)
- [Self-Supervised Contrastive Learning for Unsupervised Phoneme Segmentation](https://mlsanity.com/paper/2007.13465/) (2020)
- [The Zero Resource Speech Challenge 2019: TTS without T](https://mlsanity.com/paper/1904.11469/) (2019)
- [Unsupervised Pre-training of Bidirectional Speech Encoders via Masked Reconstruction](https://mlsanity.com/paper/2001.10603/) (2020)
- [An embedded segmental K-means model for unsupervised segmentation and clustering of speech](https://mlsanity.com/paper/1703.08135/) (2017)
- [Truly unsupervised acoustic word embeddings using weak top-down constraints in encoder-decoder models](https://mlsanity.com/paper/1811.00403/) (2018)
- [Deep convolutional acoustic word embeddings using word-pair side information](https://mlsanity.com/paper/1510.01032/) (2015)
- [Discriminative Acoustic Word Embeddings: Recurrent Neural Network-Based Approaches](https://mlsanity.com/paper/1611.02550/) (2016)
- [Evaluating the reliability of acoustic speech embeddings](https://mlsanity.com/paper/2007.13542/) (2020)
- [Multi-view Recurrent Neural Acoustic Word Embeddings](https://mlsanity.com/paper/1611.04496/) (2016)
- [End-to-End ASR-free Keyword Search from Speech](https://mlsanity.com/paper/1701.04313/) (2017)
- [Segmental Audio Word2Vec: Representing Utterances as Sequences of Vectors with Applications in Spoken Term Detection](https://mlsanity.com/paper/1808.02228/) (2018)
- [Multilingual Jointly Trained Acoustic and Written Word Embeddings](https://mlsanity.com/paper/2006.14007/) (2020)
- [Unsupervised Discovery of Recurring Speech Patterns Using Probabilistic Adaptive Metrics](https://mlsanity.com/paper/2008.00731/) (2020)
- [Audio-Linguistic Embeddings for Spoken Sentences](https://mlsanity.com/paper/1902.07817/) (2019)
- [Acoustically Grounded Word Embeddings for Improved Acoustics-to-Word Speech Recognition](https://mlsanity.com/paper/1903.12306/) (2019)
- [Phonetic-and-Semantic Embedding of Spoken Words with Applications in Spoken Content Retrieval](https://mlsanity.com/paper/1807.08089/) (2018)
- [Improved acoustic word embeddings for zero-resource languages using multilingual transfer](https://mlsanity.com/paper/2006.02295/) (2020)

## Built on by (most cited first, 19 of 19)

- [Layer-wise Analysis of a Self-supervised Speech Representation Model](https://mlsanity.com/paper/2107.04734/) (2021)
- [Language-Routing Mixture of Experts for Multilingual and Code-Switching Speech Recognition](https://mlsanity.com/paper/2307.05956/) (2023)
- [LAE: Language-Aware Encoder for Monolingual and Multilingual ASR](https://mlsanity.com/paper/2206.02093/) (2022)
- [Analyzing Acoustic Word Embeddings from Pre-trained Self-supervised Speech Models](https://mlsanity.com/paper/2210.16043/) (2022)
- [Generative Spoken Language Model based on continuous word-sized audio tokens](https://mlsanity.com/paper/2310.05224/) (2023)
- [DP-Parse: Finding Word Boundaries from Raw Speech with an Instance Lexicon](https://mlsanity.com/paper/2206.11332/) (2022)
- [A Brief Overview of Unsupervised Neural Speech Representation Learning](https://mlsanity.com/paper/2203.01829/) (2022)
- [Multilingual transfer of acoustic word embeddings improves when training on languages related to the target zero-resource language](https://mlsanity.com/paper/2106.12834/) (2021)
- [Speech Sequence Embeddings using Nearest Neighbors Contrastive Learning](https://mlsanity.com/paper/2204.05148/) (2022)
- [Discovering Phonetic Inventories with Crosslingual Automatic Speech Recognition](https://mlsanity.com/paper/2201.11207/) (2022)
- [Towards hate speech detection in low-resource languages: Comparing ASR to acoustic word embeddings on Wolof and Swahili](https://mlsanity.com/paper/2306.00410/) (2023)
- [Visually grounded few-shot word learning in low-resource settings](https://mlsanity.com/paper/2306.11371/) (2023)
- [Analyzing the Representational Geometry of Acoustic Word Embeddings](https://mlsanity.com/paper/2301.03012/) (2023)
- [Acoustic Word Embeddings for Untranscribed Target Languages with Continued Pretraining and Learned Pooling](https://mlsanity.com/paper/2306.02153/) (2023)
- [Leveraging multilingual transfer for unsupervised semantic acoustic word embeddings](https://mlsanity.com/paper/2307.02083/) (2023)
- [Supervised Acoustic Embeddings And Their Transferability Across Languages](https://mlsanity.com/paper/2301.01020/) (2023)
- [Towards few-shot isolated word reading assessment](https://mlsanity.com/paper/2507.12217/) (2025)
- [Joint Multimodal Contrastive Learning for Robust Spoken Term Detection and Keyword Spotting](https://mlsanity.com/paper/2512.14115/) (2025)
- [Low resource cross-modal alignment using HGNN to enhance speech representation](https://mlsanity.com/paper/2609.23191/) (2026)

## Abstract (arXiv)

Acoustic word embeddings (AWEs) are fixed-dimensional representations of variable-length speech segments. For zero-resource languages where labelled data is not available, one AWE approach is to use unsupervised autoencoder-based recurrent models. Another recent approach is to use multilingual transfer: a supervised AWE model is trained on several well-resourced languages and then applied to an unseen zero-resource language. We consider how a recent contrastive learning loss can be used in both the purely unsupervised and multilingual transfer settings. Firstly, we show that terms from an unsupervised term discovery system can be used for contrastive self-supervision, resulting in improvements over previous unsupervised monolingual AWE models. Secondly, we consider how multilingual AWE models can be adapted to a specific zero-resource language using discovered terms. We find that self-supervised contrastive adaptation outperforms adapted multilingual correspondence autoencoder and Siamese AWE models, giving the best overall results in a word discrimination task on six zero-resource languages.