# Unsupervised Speech Recognition via Segmental Empirical Output Distribution Matching

arXiv: https://arxiv.org/abs/1812.09323 · HTML: https://mlsanity.com/paper/1812.09323/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Dec 2018.

## Built on by (most cited first, 20 of 20)

- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [Unsupervised Speech Recognition](https://mlsanity.com/paper/2105.11084/) (2021)
- [Towards End-to-end Unsupervised Speech Recognition](https://mlsanity.com/paper/2204.02492/) (2022)
- [Unsupervised Text-to-Speech Synthesis by Unsupervised Automatic Speech Recognition](https://mlsanity.com/paper/2203.15796/) (2022)
- [Variable-rate hierarchical CPC leads to acoustic unit discovery in speech](https://mlsanity.com/paper/2206.02211/) (2022)
- [Maestro-U: Leveraging joint speech-text representation learning for zero supervised speech ASR](https://mlsanity.com/paper/2210.10027/) (2022)
- [Simple and Effective Unsupervised Speech Translation](https://mlsanity.com/paper/2210.10191/) (2022)
- [Learning from Flawed Data: Weakly Supervised Automatic Speech Recognition](https://mlsanity.com/paper/2309.15796/) (2023)
- [Unsupervised Automatic Speech Recognition: A Review](https://mlsanity.com/paper/2106.04897/) (2021)
- [EURO: ESPnet Unsupervised ASR Open-source Toolkit](https://mlsanity.com/paper/2211.17196/) (2022)
- [Unsupervised Speech Recognition with N-Skipgram and Positional Unigram Matching](https://mlsanity.com/paper/2310.02382/) (2023)
- [A Theory of Unsupervised Speech Recognition](https://mlsanity.com/paper/2306.07926/) (2023)
- [REBORN: Reinforcement-Learned Boundary Segmentation with Iterative Training for Unsupervised ASR](https://mlsanity.com/paper/2402.03988/) (2024)
- [Bridging Speech and Textual Pre-trained Models with Unsupervised ASR](https://mlsanity.com/paper/2211.03025/) (2022)
- [Word Order Does Not Matter For Speech Recognition](https://mlsanity.com/paper/2110.05994/) (2021)
- [The Performance Evaluation of Attention-Based Neural ASR under Mixed Speech Input](https://mlsanity.com/paper/2108.01245/) (2021)
- [Learning Phone Recognition from Unpaired Audio and Phone Sequences Based on Generative Adversarial Network](https://mlsanity.com/paper/2207.14568/) (2022)
- [Towards Matching Phones and Speech Representations](https://mlsanity.com/paper/2310.17558/) (2023)
- [Towards Unsupervised Speech Recognition Without Pronunciation Models](https://mlsanity.com/paper/2406.08380/) (2024)
- [Sequence-Level Unsupervised Training in Speech Recognition: A Theoretical Study](https://mlsanity.com/paper/2603.02285/) (2026)

## Abstract (arXiv)

We consider the problem of training speech recognition systems without using any labeled data, under the assumption that the learner can only access to the input utterances and a phoneme language model estimated from a non-overlapping corpus. We propose a fully unsupervised learning algorithm that alternates between solving two sub-problems: (i) learn a phoneme classifier for a given set of phoneme segmentation boundaries, and (ii) refining the phoneme boundaries based on a given classifier. To solve the first sub-problem, we introduce a novel unsupervised cost function named Segmental Empirical Output Distribution Matching, which generalizes the work in (Liu et al., 2017) to segmental structures. For the second sub-problem, we develop an approximate MAP approach to refining the boundaries obtained from Wang et al. (2017). Experimental results on TIMIT dataset demonstrate the success of this fully unsupervised phoneme recognition system, which achieves a phone error rate (PER) of 41.6%. Although it is still far away from the state-of-the-art supervised systems, we show that with oracle boundaries and matching language model, the PER could be improved to 32.5%.This performance approaches the supervised system of the same model architecture, demonstrating the great potential of the proposed method.