# Unsupervised Automatic Speech Recognition: A Review

arXiv: https://arxiv.org/abs/2106.04897 · HTML: https://mlsanity.com/paper/2106.04897/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 147 references that are in the map.
- First posted on arXiv 9 Jun 2021.

## Builds on (most cited first, 30 of 52)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [A Survey on Contrastive Self-supervised Learning](https://mlsanity.com/paper/2011.00362/) (2020)
- [Word Translation Without Parallel Data](https://mlsanity.com/paper/1710.04087/) (2017)
- [Libri-Light: A Benchmark for ASR with Limited or No Supervision](https://mlsanity.com/paper/1912.07875/) (2019)
- [Deep Speech 2: End-to-End Speech Recognition in English and Mandarin](https://mlsanity.com/paper/1512.02595/) (2015)
- [State-of-the-art Speech Recognition With Sequence-to-Sequence Models](https://mlsanity.com/paper/1712.01769/) (2017)
- [An Unsupervised Autoregressive Model for Speech Representation Learning](https://mlsanity.com/paper/1904.03240/) (2019)
- [Unsupervised Speech Recognition](https://mlsanity.com/paper/2105.11084/) (2021)
- [A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings](https://mlsanity.com/paper/1805.06297/) (2018)
- [End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures](https://mlsanity.com/paper/1911.08460/) (2019)
- [Hierarchical Multiscale Recurrent Neural Networks](https://mlsanity.com/paper/1609.01704/) (2016)
- [Unsupervised pretraining transfers well across languages](https://mlsanity.com/paper/2002.02848/) (2020)
- [Effectiveness of self-supervised pre-training for speech recognition](https://mlsanity.com/paper/1911.03912/) (2019)
- [Speech Recognition Challenge in the Wild: Arabic MGB-3](https://mlsanity.com/paper/1709.07276/) (2017)
- [Deep Multimodal Semantic Embeddings for Speech and Images](https://mlsanity.com/paper/1511.03690/) (2015)
- [Vector-quantized neural networks for acoustic unit discovery in the ZeroSpeech 2020 challenge](https://mlsanity.com/paper/2005.09409/) (2020)
- [Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speech](https://mlsanity.com/paper/1803.08976/) (2018)
- [The Zero Resource Speech Challenge 2017](https://mlsanity.com/paper/1712.04313/) (2017)
- [Wav2Letter: an End-to-End ConvNet-based Speech Recognition System](https://mlsanity.com/paper/1609.03193/) (2016)
- [Self-Supervised Contrastive Learning for Unsupervised Phoneme Segmentation](https://mlsanity.com/paper/2007.13465/) (2020)
- [The Zero Resource Speech Challenge 2020: Discovering discrete subword and word units](https://mlsanity.com/paper/2010.05967/) (2020)
- [The Zero Resource Speech Challenge 2019: TTS without T](https://mlsanity.com/paper/1904.11469/) (2019)
- [Representations of language in a model of visually grounded speech signal](https://mlsanity.com/paper/1702.01991/) (2017)
- [Audio Word2Vec: Unsupervised Learning of Audio Segment Representations using Sequence-to-sequence Autoencoder](https://mlsanity.com/paper/1603.00982/) (2016)
- [Unsupervised Cross-Modal Alignment of Speech and Text Embedding Spaces](https://mlsanity.com/paper/1805.07467/) (2018)

## Built on by (most cited first, 5 of 5)

- [Automatic Speech Recognition using Advanced Deep Learning Approaches: A survey](https://mlsanity.com/paper/2403.01255/) (2024)
- [NoRefER: a Referenceless Quality Metric for Automatic Speech Recognition via Semi-Supervised Language Model Fine-Tuning with Contrastive Learning](https://mlsanity.com/paper/2306.12577/) (2023)
- [Supervised Acoustic Embeddings And Their Transferability Across Languages](https://mlsanity.com/paper/2301.01020/) (2023)
- [Unsupervised ASR via Cross-Lingual Pseudo-Labeling](https://mlsanity.com/paper/2305.13330/) (2023)
- [Spoken Word2Vec: Learning Skipgram Embeddings from Speech](https://mlsanity.com/paper/2311.09319/) (2023)

## Abstract (arXiv)

Automatic Speech Recognition (ASR) systems can be trained to achieve remarkable performance given large amounts of manually transcribed speech, but large labeled data sets can be difficult or expensive to acquire for all languages of interest. In this paper, we review the research literature to identify models and ideas that could lead to fully unsupervised ASR, including unsupervised segmentation of the speech signal, unsupervised mapping from speech segments to text, and semi-supervised models with nominal amounts of labeled examples. The objective of the study is to identify the limitations of what can be learned from speech data alone and to understand the minimum requirements for speech recognition. Identifying these limitations would help optimize the resources and efforts in ASR development for low-resource languages.