# Phone-to-audio alignment without text: A Semi-supervised Approach

arXiv: https://arxiv.org/abs/2110.03876 · HTML: https://mlsanity.com/paper/2110.03876/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 31 references that are in the map.
- First posted on arXiv 8 Oct 2021.

## Builds on (most cited first, 15 of 15)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [Datasets: A Community Library for Natural Language Processing](https://mlsanity.com/paper/2109.02846/) (2021)
- [Speech Model Pre-training for End-to-End Spoken Language Understanding](https://mlsanity.com/paper/1904.03670/) (2019)
- [Improved Noisy Student Training for Automatic Speech Recognition](https://mlsanity.com/paper/2005.09629/) (2020)
- [CTC-Segmentation of Large Corpora for German End-to-end Speech Recognition](https://mlsanity.com/paper/2007.09127/) (2020)
- [Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attention](https://mlsanity.com/paper/1710.08969/) (2017)
- [One TTS Alignment To Rule Them All](https://mlsanity.com/paper/2108.10447/) (2021)
- [Self-Supervised Contrastive Learning for Unsupervised Phoneme Segmentation](https://mlsanity.com/paper/2007.13465/) (2020)
- [Segmental Contrastive Predictive Coding for Unsupervised Word Segmentation](https://mlsanity.com/paper/2106.02170/) (2021)
- [Towards unsupervised phone and word segmentation using self-supervised vector-quantized neural networks](https://mlsanity.com/paper/2012.07551/) (2020)

## Built on by (most cited first, 26 of 26)

- [EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions](https://mlsanity.com/paper/2409.18042/) (2024)
- [Encoding of lexical tone in self-supervised models of spoken language](https://mlsanity.com/paper/2403.16865/) (2024)
- [The taste of IPA: Towards open-vocabulary keyword spotting and forced alignment in any language](https://mlsanity.com/paper/2311.08323/) (2023)
- [Less Peaky and More Accurate CTC Forced Alignment by Label Priors](https://mlsanity.com/paper/2406.02560/) (2024)
- [MultiPA: A Multi-task Speech Pronunciation Assessment Model for Open Response Scenarios](https://mlsanity.com/paper/2308.12490/) (2023)
- [Weakly-supervised forced alignment of disfluent speech using phoneme-level modeling](https://mlsanity.com/paper/2306.00996/) (2023)
- [Audio2Face-3D: Audio-driven Realistic Facial Animation For Digital Avatars](https://mlsanity.com/paper/2508.16401/) (2025)
- [PSST! Prosodic Speech Segmentation with Transformers](https://mlsanity.com/paper/2302.01984/) (2023)
- [Seamless Dysfluent Speech Text Alignment for Disordered Speech Analysis](https://mlsanity.com/paper/2506.12073/) (2025)
- [Multimodal Lyrics-Rhythm Matching](https://mlsanity.com/paper/2301.02732/) (2023)
- [PDAF: A Phonetic Debiasing Attention Framework For Speaker Verification](https://mlsanity.com/paper/2409.05799/) (2024)
- [Montreal Forced Aligner and the state of speech-to-text alignment in 2026](https://mlsanity.com/paper/2606.18466/) (2026)
- [PAAPLoss: A Phonetic-Aligned Acoustic Parameter Loss for Speech Enhancement](https://mlsanity.com/paper/2302.08095/) (2023)
- [The complementary roles of non-verbal cues for Robust Pronunciation Assessment](https://mlsanity.com/paper/2309.07739/) (2023)
- [K-Function: Joint Pronunciation Transcription and Feedback for Evaluating Kids Language Function](https://mlsanity.com/paper/2507.03043/) (2025)
- [How Far Do SSL Speech Models Listen for Tone? Temporal Focus of Tone Representation under Low-resource Transfer](https://mlsanity.com/paper/2511.12285/) (2025)
- [Integrating Human Linguistic Insights into AI: Theory-Driven Representation for Multilingual Text-to-Speech](https://mlsanity.com/paper/2204.07228/) (2022)
- [TIPAA-SSL: Text Independent Phone-to-Audio Alignment based on Self-Supervised Learning and Knowledge Transfer](https://mlsanity.com/paper/2405.02124/) (2024)
- [Speaker- and Text-Independent Estimation of Articulatory Movements and Phoneme Alignments from Speech](https://mlsanity.com/paper/2407.03132/) (2024)
- [Gradient boundaries through confidence intervals for forced alignment estimates using model ensembles](https://mlsanity.com/paper/2506.01256/) (2025)
- [wav2VOT: Automatic estimation of voice onset time, closure duration, and burst realisation with wav2vec2](https://mlsanity.com/paper/2606.28857/) (2026)
- [Phone Segmentation and Recognition through Phonological Activation Mapping](https://mlsanity.com/paper/2607.09020/) (2026)
- [Phonetic forced alignment for low-resource language varieties: Model training and evaluation on Chengdu Mandarin](https://mlsanity.com/paper/2607.21332/) (2026)
- [FA-Bench: A Benchmark for Phone- and Word-Level Timestamp Accuracy in Forced Alignment and ASR on Clean and Noisy Speech](https://mlsanity.com/paper/2609.32396/) (2026)
- [FuseAlign: Forced Alignment in the Wild](https://mlsanity.com/paper/2609.33650/) (2026)
- [Where Words Begin and End](https://mlsanity.com/paper/2610.17782/) (2026)

## Abstract (arXiv)

The task of phone-to-audio alignment has many applications in speech research. Here we introduce two Wav2Vec2-based models for both text-dependent and text-independent phone-to-audio alignment. The proposed Wav2Vec2-FS, a semi-supervised model, directly learns phone-to-audio alignment through contrastive learning and a forward sum loss, and can be coupled with a pretrained phone recognizer to achieve text-independent alignment. The other model, Wav2Vec2-FC, is a frame classification model trained on forced aligned labels that can both perform forced alignment and text-independent segmentation. Evaluation results suggest that both proposed methods, even when transcriptions are not available, generate highly close results to existing forced alignment tools. Our work presents a neural pipeline of fully automated phone-to-audio alignment. Code and pretrained models are available at https://github.com/lingjzhu/charsiu.