# Simple and Effective Zero-shot Cross-lingual Phoneme Recognition

arXiv: https://arxiv.org/abs/2109.11680 · HTML: https://mlsanity.com/paper/2109.11680/

## Key facts

- Cited by 79 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 39 references that are in the map.
- First posted on arXiv 23 Sep 2021.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [MLS: A Large-Scale Multilingual Dataset for Speech Research](https://mlsanity.com/paper/2012.03411/) (2020)
- [Libri-Light: A Benchmark for ASR with Limited or No Supervision](https://mlsanity.com/paper/1912.07875/) (2019)
- [vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations](https://mlsanity.com/paper/1910.05453/) (2019)
- [An Unsupervised Autoregressive Model for Speech Representation Learning](https://mlsanity.com/paper/1904.03240/) (2019)
- [Unsupervised Speech Recognition](https://mlsanity.com/paper/2105.11084/) (2021)
- [Improved Noisy Student Training for Automatic Speech Recognition](https://mlsanity.com/paper/2005.09629/) (2020)
- [End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures](https://mlsanity.com/paper/1911.08460/) (2019)
- [Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameters](https://mlsanity.com/paper/2007.03001/) (2020)
- [Self-training and Pre-training are Complementary for Speech Recognition](https://mlsanity.com/paper/2010.11430/) (2020)
- [Iterative Pseudo-Labeling for Speech Recognition](https://mlsanity.com/paper/2005.09267/) (2020)
- [Universal Phone Recognition with a Multilingual Allophone System](https://mlsanity.com/paper/2002.11800/) (2020)
- [The challenge of realistic music generation: modelling raw audio at scale](https://mlsanity.com/paper/1806.10474/) (2018)
- [SlimIPL: Language-Model-Free Iterative Pseudo-Labeling](https://mlsanity.com/paper/2010.11524/) (2020)
- [Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speech](https://mlsanity.com/paper/1803.08976/) (2018)
- [wav2letter++: The Fastest Open-source Speech Recognition System](https://mlsanity.com/paper/1812.07625/) (2018)
- [Adapt-and-Adjust: Overcoming the Long-Tail Problem of Multilingual Speech Recognition](https://mlsanity.com/paper/2012.01687/) (2020)
- [Completely Unsupervised Phoneme Recognition by Adversarially Learning Mapping Relationships from Audio Embeddings](https://mlsanity.com/paper/1804.00316/) (2018)
- [Sequence-based Multi-lingual Low Resource Speech Recognition](https://mlsanity.com/paper/1802.07420/) (2018)
- [Towards Zero-shot Learning for Automatic Phonemic Transcription](https://mlsanity.com/paper/2002.11781/) (2020)
- [Differentiable Allophone Graphs for Language-Universal Speech Recognition](https://mlsanity.com/paper/2107.11628/) (2021)
- [Multilingual transfer of acoustic word embeddings improves when training on languages related to the target zero-resource language](https://mlsanity.com/paper/2106.12834/) (2021)

## Built on by (most cited first, 30 of 79)

- [GaussianTalker: Speaker-specific Talking Head Synthesis via 3D Gaussian Splatting](https://mlsanity.com/paper/2404.14037/) (2024)
- [ZIPA: A family of efficient models for multilingual phone recognition](https://mlsanity.com/paper/2505.23170/) (2025)
- [Adapting Multilingual Speech Representation Model for a New, Underresourced Language through Multilingual Fine-tuning and Continued Pretraining](https://mlsanity.com/paper/2301.07295/) (2023)
- [Whistle: Data-Efficient Multilingual and Crosslingual Speech Recognition via Weakly Phonetic Supervision](https://mlsanity.com/paper/2406.02166/) (2024)
- [Universal Automatic Phonetic Transcription into the International Phonetic Alphabet](https://mlsanity.com/paper/2308.03917/) (2023)
- [The taste of IPA: Towards open-vocabulary keyword spotting and forced alignment in any language](https://mlsanity.com/paper/2311.08323/) (2023)
- [POWSM: A Phonetic Open Whisper-Style Speech Foundation Model](https://mlsanity.com/paper/2510.24992/) (2025)
- [Allophant: Cross-lingual Phoneme Recognition with Articulatory Attributes](https://mlsanity.com/paper/2306.04306/) (2023)
- [[b] = [d] - [t] + [p]: Self-supervised Speech Models Discover Phonological Vector Arithmetic](https://mlsanity.com/paper/2602.18899/) (2026)
- [Combining Spectral and Self-Supervised Features for Low Resource Speech Recognition and Translation](https://mlsanity.com/paper/2204.02470/) (2022)
- [DurFlex-EVC: Duration-Flexible Emotional Voice Conversion Leveraging Discrete Representations without Text Alignment](https://mlsanity.com/paper/2401.08095/) (2024)
- [Large Language Models for Dysfluency Detection in Stuttered Speech](https://mlsanity.com/paper/2406.11025/) (2024)
- [Out of Thin Air: Is Zero-Shot Cross-Lingual Keyword Detection Better Than Unsupervised?](https://mlsanity.com/paper/2202.06650/) (2022)
- [PANCETTA: Phoneme Aware Neural Completion to Elicit Tongue Twisters Automatically](https://mlsanity.com/paper/2209.06275/) (2022)
- [READ Avatars: Realistic Emotion-controllable Audio Driven Avatars](https://mlsanity.com/paper/2303.00744/) (2023)
- [Multilingual Word Error Rate Estimation: e-WER3](https://mlsanity.com/paper/2304.00649/) (2023)
- [SelfVC: Voice Conversion With Iterative Refinement using Self Transformations](https://mlsanity.com/paper/2310.09653/) (2023)
- [Optimizing Two-Pass Cross-Lingual Transfer Learning: Phoneme Recognition and Phoneme to Grapheme Translation](https://mlsanity.com/paper/2312.03312/) (2023)
- [Improving Cross-Lingual Phonetic Representation of Low-Resource Languages Through Language Similarity Analysis](https://mlsanity.com/paper/2501.06810/) (2025)
- [Dysfluent WFST: A Framework for Zero-Shot Speech Dysfluency Transcription and Detection](https://mlsanity.com/paper/2505.16351/) (2025)
- [PIA: Deepfake Detection Using Phoneme-Temporal and Identity-Dynamic Analysis](https://mlsanity.com/paper/2510.14241/) (2025)
- [An Empirical Recipe for Universal Phone Recognition](https://mlsanity.com/paper/2603.29042/) (2026)
- [Applying wav2vec2 for Speech Recognition on Bengali Common Voices Dataset](https://mlsanity.com/paper/2209.06581/) (2022)
- [Uncertainty as a Predictor: Leveraging Self-Supervised Learning for Zero-Shot MOS Prediction](https://mlsanity.com/paper/2312.15616/) (2023)
- [Articulatory Feature Prediction from Surface EMG during Speech Production](https://mlsanity.com/paper/2505.13814/) (2025)
- [Seamless Dysfluent Speech Text Alignment for Disordered Speech Analysis](https://mlsanity.com/paper/2506.12073/) (2025)
- [PRiSM: Benchmarking Phone Realization in Speech Models](https://mlsanity.com/paper/2601.14046/) (2026)
- [Quantifying Speaker Embedding Phonological Rule Interactions in Accented Speech Synthesis](https://mlsanity.com/paper/2601.14417/) (2026)
- [Multilingual Dysarthric Speech Assessment Using Universal Phone Recognition and Language-Specific Phonemic Contrast Modeling](https://mlsanity.com/paper/2601.21205/) (2026)
- [Automatic Speech Recognition (ASR) for the Diagnosis of pronunciation of Speech Sound Disorders in Korean children](https://mlsanity.com/paper/2403.08187/) (2024)

## Abstract (arXiv)

Recent progress in self-training, self-supervised pretraining and unsupervised learning enabled well performing speech recognition systems without any labeled data. However, in many cases there is labeled data available for related languages which is not utilized by these methods. This paper extends previous work on zero-shot cross-lingual transfer learning by fine-tuning a multilingually pretrained wav2vec 2.0 model to transcribe unseen languages. This is done by mapping phonemes of the training languages to the target language using articulatory features. Experiments show that this simple method significantly outperforms prior work which introduced task-specific architectures and used only part of a monolingually pretrained model.