# Leveraging pre-trained representations to improve access to untranscribed speech from endangered languages

arXiv: https://arxiv.org/abs/2103.14583 · HTML: https://mlsanity.com/paper/2103.14583/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 22 references that are in the map.
- First posted on arXiv 26 Mar 2021.

## Builds on (most cited first, 8 of 8)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations](https://mlsanity.com/paper/1910.05453/) (2019)
- [Self-Training for End-to-End Speech Recognition](https://mlsanity.com/paper/1909.09116/) (2019)
- [Neural Network based End-to-End Query by Example Spoken Term Detection](https://mlsanity.com/paper/1911.08332/) (2019)
- [Almost-unsupervised Speech Recognition with Close-to-zero Resource Based on Phonetic Structures Learned from Very Small Unpaired Speech and Text Data](https://mlsanity.com/paper/1810.12566/) (2018)
- [Enabling Interactive Transcription in an Indigenous Community](https://mlsanity.com/paper/2011.06198/) (2020)

## Built on by (most cited first, 8 of 8)

- [Making More of Little Data: Improving Low-Resource Automatic Speech Recognition Using Data Augmentation](https://mlsanity.com/paper/2305.10951/) (2023)
- [Automated speech tools for helping communities process restricted-access corpora for language revival efforts](https://mlsanity.com/paper/2204.07272/) (2022)
- [Analyzing the Representational Geometry of Acoustic Word Embeddings](https://mlsanity.com/paper/2301.03012/) (2023)
- [From `Snippet-lects' to Doculects and Dialects: Leveraging Neural Representations of Speech for Placing Audio Signals in a Language Landscape](https://mlsanity.com/paper/2305.18602/) (2023)
- [Efficiency-oriented approaches for self-supervised speech representation learning](https://mlsanity.com/paper/2312.11142/) (2023)
- [Data Augmentation and Hyperparameter Tuning for Low-Resource MFA](https://mlsanity.com/paper/2504.07024/) (2025)
- [Multilingual MFA: Forced Alignment on Low-Resource Related Languages](https://mlsanity.com/paper/2504.07315/) (2025)
- [Investigating self-supervised, weakly supervised and fully supervised training approaches for multi-domain automatic speech recognition: a study on Bangladeshi Bangla](https://mlsanity.com/paper/2210.12921/) (2022)

## Abstract (arXiv)

Pre-trained speech representations like wav2vec 2.0 are a powerful tool for automatic speech recognition (ASR). Yet many endangered languages lack sufficient data for pre-training such models, or are predominantly oral vernaculars without a standardised writing system, precluding fine-tuning. Query-by-example spoken term detection (QbE-STD) offers an alternative for iteratively indexing untranscribed speech corpora by locating spoken query terms. Using data from 7 Australian Aboriginal languages and a regional variety of Dutch, all of which are endangered or vulnerable, we show that QbE-STD can be improved by leveraging representations developed for ASR (wav2vec 2.0: the English monolingual model and XLSR53 multilingual model). Surprisingly, the English model outperformed the multilingual model on 4 Australian language datasets, raising questions around how to optimally leverage self-supervised speech representations for QbE-STD. Nevertheless, we find that wav2vec 2.0 representations (either English or XLSR53) offer large improvements (56-86% relative) over state-of-the-art approaches on our endangered language datasets.