# Multilingual transfer of acoustic word embeddings improves when training on languages related to the target zero-resource language

arXiv: https://arxiv.org/abs/2106.12834 · HTML: https://mlsanity.com/paper/2106.12834/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 44 references that are in the map.
- First posted on arXiv 24 Jun 2021.

## Builds on (most cited first, 30 of 31)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speech](https://mlsanity.com/paper/1803.08976/) (2018)
- [Masakhane -- Machine Translation For Africa](https://mlsanity.com/paper/2003.11529/) (2020)
- [Self-Supervised Contrastive Learning for Unsupervised Phoneme Segmentation](https://mlsanity.com/paper/2007.13465/) (2020)
- [An embedded segmental K-means model for unsupervised segmentation and clustering of speech](https://mlsanity.com/paper/1703.08135/) (2017)
- [Truly unsupervised acoustic word embeddings using weak top-down constraints in encoder-decoder models](https://mlsanity.com/paper/1811.00403/) (2018)
- [Deep convolutional acoustic word embeddings using word-pair side information](https://mlsanity.com/paper/1510.01032/) (2015)
- [Query-by-Example Search with Discriminative Neural Acoustic Word Embeddings](https://mlsanity.com/paper/1706.03818/) (2017)
- [Evaluating the reliability of acoustic speech embeddings](https://mlsanity.com/paper/2007.13542/) (2020)
- [Multi-view Recurrent Neural Acoustic Word Embeddings](https://mlsanity.com/paper/1611.04496/) (2016)
- [End-to-End ASR-free Keyword Search from Speech](https://mlsanity.com/paper/1701.04313/) (2017)
- [Acoustic word embeddings for zero-resource languages using self-supervised contrastive learning and multilingual adaptation](https://mlsanity.com/paper/2103.10731/) (2021)
- [Segmental Audio Word2Vec: Representing Utterances as Sequences of Vectors with Applications in Spoken Term Detection](https://mlsanity.com/paper/1808.02228/) (2018)
- [Multilingual Jointly Trained Acoustic and Written Word Embeddings](https://mlsanity.com/paper/2006.14007/) (2020)
- [Unsupervised Discovery of Recurring Speech Patterns Using Probabilistic Adaptive Metrics](https://mlsanity.com/paper/2008.00731/) (2020)
- [Audio-Linguistic Embeddings for Spoken Sentences](https://mlsanity.com/paper/1902.07817/) (2019)
- [Acoustically Grounded Word Embeddings for Improved Acoustics-to-Word Speech Recognition](https://mlsanity.com/paper/1903.12306/) (2019)
- [Phonetic-and-Semantic Embedding of Spoken Words with Applications in Spoken Content Retrieval](https://mlsanity.com/paper/1807.08089/) (2018)
- [Semantic query-by-example speech search using visual grounding](https://mlsanity.com/paper/1904.07078/) (2019)
- [Improved acoustic word embeddings for zero-resource languages using multilingual transfer](https://mlsanity.com/paper/2006.02295/) (2020)
- [Bayesian Subspace Hidden Markov Model for Acoustic Unit Discovery](https://mlsanity.com/paper/1904.03876/) (2019)
- [Multilingual acoustic word embedding models for processing zero-resource languages](https://mlsanity.com/paper/2002.02109/) (2020)
- [Learning Acoustic Word Embeddings with Temporal Context for Query-by-Example Speech Search](https://mlsanity.com/paper/1806.03621/) (2018)
- [Acoustic span embeddings for multilingual query-by-example search](https://mlsanity.com/paper/2011.11807/) (2020)
- [Multilingual and Unsupervised Subword Modeling for Zero-Resource Languages](https://mlsanity.com/paper/1811.04791/) (2018)
- [Additional Shared Decoder on Siamese Multi-view Encoders for Learning Acoustic Word Embeddings](https://mlsanity.com/paper/1910.00341/) (2019)
- [Contextual Joint Factor Acoustic Embeddings](https://mlsanity.com/paper/1910.07601/) (2019)
- [Improved Audio Embeddings by Adjacency-Based Clustering with Applications in Spoken Term Detection](https://mlsanity.com/paper/1811.02775/) (2018)
- [SylNet: An Adaptable End-to-End Syllable Count Estimator for Speech](https://mlsanity.com/paper/1906.09825/) (2019)

## Built on by (most cited first, 7 of 7)

- [Simple and Effective Zero-shot Cross-lingual Phoneme Recognition](https://mlsanity.com/paper/2109.11680/) (2021)
- [How Familiar Does That Sound? Cross-Lingual Representational Similarity Analysis of Acoustic Word Embeddings](https://mlsanity.com/paper/2109.10179/) (2021)
- [Towards hate speech detection in low-resource languages: Comparing ASR to acoustic word embeddings on Wolof and Swahili](https://mlsanity.com/paper/2306.00410/) (2023)
- [Integrating Form and Meaning: A Multi-Task Learning Model for Acoustic Word Embeddings](https://mlsanity.com/paper/2209.06633/) (2022)
- [Acoustic Word Embeddings for Untranscribed Target Languages with Continued Pretraining and Learned Pooling](https://mlsanity.com/paper/2306.02153/) (2023)
- [Leveraging multilingual transfer for unsupervised semantic acoustic word embeddings](https://mlsanity.com/paper/2307.02083/) (2023)
- [Low resource cross-modal alignment using HGNN to enhance speech representation](https://mlsanity.com/paper/2609.23191/) (2026)

## Abstract (arXiv)

Acoustic word embedding models map variable duration speech segments to fixed dimensional vectors, enabling efficient speech search and discovery. Previous work explored how embeddings can be obtained in zero-resource settings where no labelled data is available in the target language. The current best approach uses transfer learning: a single supervised multilingual model is trained using labelled data from multiple well-resourced languages and then applied to a target zero-resource language (without fine-tuning). However, it is still unclear how the specific choice of training languages affect downstream performance. Concretely, here we ask whether it is beneficial to use training languages related to the target. Using data from eleven languages spoken in Southern Africa, we experiment with adding data from different language families while controlling for the amount of data per language. In word discrimination and query-by-example search evaluations, we show that training on languages from the same family gives large improvements. Through finer-grained analysis, we show that training on even just a single related language gives the largest gain. We also find that adding data from unrelated languages generally doesn't hurt performance.