# TIPAA-SSL: Text Independent Phone-to-Audio Alignment based on Self-Supervised Learning and Knowledge Transfer

arXiv: https://arxiv.org/abs/2405.02124 · HTML: https://mlsanity.com/paper/2405.02124/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 21 references that are in the map.
- First posted on arXiv 3 May 2024.

## Builds on (most cited first, 14 of 14)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [A Survey on Contrastive Self-supervised Learning](https://mlsanity.com/paper/2011.00362/) (2020)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [A Survey on Deep Transfer Learning](https://mlsanity.com/paper/1808.01974/) (2018)
- [Transfer Learning for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/1604.02201/) (2016)
- [Emotional Voice Conversion: Theory, Databases and ESD](https://mlsanity.com/paper/2105.14762/) (2021)
- [Simple and Effective Zero-shot Cross-lingual Phoneme Recognition](https://mlsanity.com/paper/2109.11680/) (2021)
- [Phone-to-audio alignment without text: A Semi-supervised Approach](https://mlsanity.com/paper/2110.03876/) (2021)
- [Exploring Transfer Learning for Low Resource Emotional TTS](https://mlsanity.com/paper/1901.04276/) (2019)
- [Visualization and Interpretation of Latent Spaces for Controlling Expressive Speech Synthesis through Audio Analysis](https://mlsanity.com/paper/1903.11570/) (2019)
- [ASR-based Features for Emotion Recognition: A Transfer Learning Approach](https://mlsanity.com/paper/1805.09197/) (2018)
- [Analysis and Assessment of Controllability of an Expressive Deep Learning-based TTS system](https://mlsanity.com/paper/2103.04097/) (2021)
- [Phonemic and Graphemic Multilingual CTC Based Speech Recognition](https://mlsanity.com/paper/1711.04564/) (2017)
- [Flowchase: a Mobile Application for Pronunciation Training](https://mlsanity.com/paper/2307.02051/) (2023)

## Abstract (arXiv)

In this paper, we present a novel approach for text independent phone-to-audio alignment based on phoneme recognition, representation learning and knowledge transfer. Our method leverages a self-supervised model (wav2vec2) fine-tuned for phoneme recognition using a Connectionist Temporal Classification (CTC) loss, a dimension reduction model and a frame-level phoneme classifier trained thanks to forced-alignment labels (using Montreal Forced Aligner) to produce multi-lingual phonetic representations, thus requiring minimal additional training. We evaluate our model using synthetic native data from the TIMIT dataset and the SCRIBE dataset for American and British English, respectively. Our proposed model outperforms the state-of-the-art (charsiu) in statistical metrics and has applications in language learning and speech processing systems. We leave experiments on other languages for future work but the design of the system makes it easily adaptable to other languages.