# Completely Unsupervised Phoneme Recognition by Adversarially Learning Mapping Relationships from Audio Embeddings

arXiv: https://arxiv.org/abs/1804.00316 · HTML: https://mlsanity.com/paper/1804.00316/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Apr 2018.

## Built on by (most cited first, 22 of 22)

- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [Unsupervised Speech Recognition](https://mlsanity.com/paper/2105.11084/) (2021)
- [Simple and Effective Zero-shot Cross-lingual Phoneme Recognition](https://mlsanity.com/paper/2109.11680/) (2021)
- [Towards End-to-end Unsupervised Speech Recognition](https://mlsanity.com/paper/2204.02492/) (2022)
- [Whistle: Data-Efficient Multilingual and Crosslingual Speech Recognition via Weakly Phonetic Supervision](https://mlsanity.com/paper/2406.02166/) (2024)
- [Maestro-U: Leveraging joint speech-text representation learning for zero supervised speech ASR](https://mlsanity.com/paper/2210.10027/) (2022)
- [Simple and Effective Unsupervised Speech Translation](https://mlsanity.com/paper/2210.10191/) (2022)
- [Zero-AVSR: Zero-Shot Audio-Visual Speech Recognition with LLMs by Learning Language-Agnostic Speech Representations](https://mlsanity.com/paper/2503.06273/) (2025)
- [Scaling A Simple Approach to Zero-Shot Speech Recognition](https://mlsanity.com/paper/2407.17852/) (2024)
- [Pushing the Limits of Unsupervised Unit Discovery for SSL Speech Representation](https://mlsanity.com/paper/2306.08920/) (2023)
- [Unsupervised Automatic Speech Recognition: A Review](https://mlsanity.com/paper/2106.04897/) (2021)
- [Improving Cascaded Unsupervised Speech Translation with Denoising Back-translation](https://mlsanity.com/paper/2305.07455/) (2023)
- [Unsupervised Speech Recognition with N-Skipgram and Positional Unigram Matching](https://mlsanity.com/paper/2310.02382/) (2023)
- [A Theory of Unsupervised Speech Recognition](https://mlsanity.com/paper/2306.07926/) (2023)
- [REBORN: Reinforcement-Learned Boundary Segmentation with Iterative Training for Unsupervised ASR](https://mlsanity.com/paper/2402.03988/) (2024)
- [Learning the joint distribution of two sequences using little or no paired data](https://mlsanity.com/paper/2212.03232/) (2022)
- [Word Order Does Not Matter For Speech Recognition](https://mlsanity.com/paper/2110.05994/) (2021)
- [Towards Unsupervised Speech Recognition at the Syllable-Level](https://mlsanity.com/paper/2510.03639/) (2025)
- [Learning Phone Recognition from Unpaired Audio and Phone Sequences Based on Generative Adversarial Network](https://mlsanity.com/paper/2207.14568/) (2022)
- [Towards Matching Phones and Speech Representations](https://mlsanity.com/paper/2310.17558/) (2023)
- [Towards Unsupervised Speech Recognition Without Pronunciation Models](https://mlsanity.com/paper/2406.08380/) (2024)
- [Sequence-Level Unsupervised Training in Speech Recognition: A Theoretical Study](https://mlsanity.com/paper/2603.02285/) (2026)

## Abstract (arXiv)

Unsupervised discovery of acoustic tokens from audio corpora without annotation and learning vector representations for these tokens have been widely studied. Although these techniques have been shown successful in some applications such as query-by-example Spoken Term Detection (STD), the lack of mapping relationships between these discovered tokens and real phonemes have limited the down-stream applications. This paper represents probably the first attempt towards the goal of completely unsupervised phoneme recognition, or mapping audio signals to phoneme sequences without phoneme-labeled audio data. The basic idea is to cluster the embedded acoustic tokens and learn the mapping between the cluster sequences and the unknown phoneme sequences with a Generative Adversarial Network (GAN). An unsupervised phoneme recognition accuracy of 36% was achieved in the preliminary experiments.