Towards Unsupervised Speech Recognition Without Pronunciation Models
Not on the map. Why?
Abstract (arXiv)
Recent advancements in supervised automatic speech recognition (ASR) have achieved remarkable performance, largely due to the growing availability of large transcribed speech corpora. However, most languages lack sufficient paired speech and text data to effectively train these systems. In this article, we tackle the challenge of developing ASR systems without paired speech and text corpora by proposing the removal of reliance on a phoneme lexicon. We explore a new research direction: word-level unsupervised ASR, and experimentally demonstrate that an unsupervised speech recognizer can emerge from joint speech-to-speech and text-to-text masked token-infilling. Using a curated speech corpus containing a fixed number of English words, our system iteratively refines the word segmentation structure and achieves a word error rate of between 20-23%, depending on the vocabulary size, without parallel transcripts, oracle word boundaries, or a pronunciation lexicon. This innovative model surpasses the performance of previous unsupervised ASR models under the lexicon-free setting.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 35 of its 47 references that are in the map. First posted on arXiv 12 Jun 2024.
Builds on
35 of 47 references.
References · 35
- 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
- 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
- 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 601
- 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 501
- 2017Word Translation Without Parallel Datacited by 487
- 2017Unsupervised Machine Translation Using Monolingual Corpora Onlycited by 273
- 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
- 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 180
- 2022Comparative layer-wise analysis of self-supervised speech modelscited by 158
- 2017Unsupervised Neural Machine Translationcited by 149
23 more
- 2021Unsupervised Speech Recognitioncited by 148
- 2022MAESTRO: Matched Speech Text Representations through Modality Matchingcited by 79
- 2021SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Trainingcited by 69
- 2023What Do Self-Supervised Speech Models Know About Words?cited by 64
- 2022SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Datacited by 49
- 2022Towards End-to-end Unsupervised Speech Recognitioncited by 45
- 2022SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-trainingcited by 40
- 2022Word Discovery in Visually Grounded, Self-Supervised Speech Modelscited by 37
- 2022Word Segmentation on Discovered Phone Units with Dynamic Programming and Self-Supervised Scoringcited by 24
- 2018Completely Unsupervised Phoneme Recognition by Adversarially Learning Mapping Relationships from Audio Embeddingscited by 22
- 2018Unsupervised Speech Recognition via Segmental Empirical Output Distribution Matchingcited by 20
- 2021Contrastive prediction strategies for unsupervised segmentation and categorization of phonemes and wordscited by 18
- 2021Unsupervised Speech Segmentation and Variable Rate Representation Learning using Segmental Contrastive Predictive Codingcited by 14
- 2022Unsupervised Text-to-Speech Synthesis by Unsupervised Automatic Speech Recognitioncited by 14
- 2022Variable-rate hierarchical CPC leads to acoustic unit discovery in speechcited by 13
- 2022DP-Parse: Finding Word Boundaries from Raw Speech with an Instance Lexiconcited by 12
- 2021Deciphering Speech: a Zero-Resource Approach to Cross-Lingual Transfer in ASRcited by 11
- 2019Completely Unsupervised Speech Recognition By A Generative Adversarial Network Harmonized With Iteratively Refined Hidden Markov Modelscited by 8
- 2023Unsupervised Word Segmentation Using Temporal Gradient Pseudo-Labelscited by 7
- 2022EURO: ESPnet Unsupervised ASR Open-source Toolkitcited by 5
- 2023Unsupervised Speech Recognition with N-Skipgram and Positional Unigram Matchingcited by 5
- 2024REBORN: Reinforcement-Learned Boundary Segmentation with Iterative Training for Unsupervised ASRcited by 4
- 2023XLS-R fine-tuning on noisy word boundaries for unsupervised speech segmentation into wordscited by 1
Built on by
Citing papers per year: 2025: 1.
1 paper.