mlsanityData to 14 Oct 2026

Phone-to-audio alignment without text: A Semi-supervised Approach

arXiv 2110.03876 ↗First posted 8 Oct 2021cs.CL cs.SD eess.AScited by 26

Not on the map. Why?

Abstract (arXiv)

The task of phone-to-audio alignment has many applications in speech research. Here we introduce two Wav2Vec2-based models for both text-dependent and text-independent phone-to-audio alignment. The proposed Wav2Vec2-FS, a semi-supervised model, directly learns phone-to-audio alignment through contrastive learning and a forward sum loss, and can be coupled with a pretrained phone recognizer to achieve text-independent alignment. The other model, Wav2Vec2-FC, is a frame classification model trained on forced aligned labels that can both perform forced alignment and text-independent segmentation. Evaluation results suggest that both proposed methods, even when transcriptions are not available, generate highly close results to existing forced alignment tools. Our work presents a neural pipeline of fully automated phone-to-audio alignment. Code and pretrained models are available at https://github.com/lingjzhu/charsiu.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Builds on 15 of its 31 references that are in the map. First posted on arXiv 8 Oct 2021.

Builds on

15 of 31 references.

References · 15

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  3. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  4. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  5. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  6. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
  7. 2021Datasets: A Community Library for Natural Language Processingcited by 530
  8. 2019Speech Model Pre-training for End-to-End Spoken Language Understandingcited by 193
  9. 2020Improved Noisy Student Training for Automatic Speech Recognitioncited by 141
  10. 2020CTC-Segmentation of Large Corpora for German End-to-end Speech Recognitioncited by 76
  11. 2017Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attentioncited by 57
  12. 2021One TTS Alignment To Rule Them Allcited by 47
3 more
  1. 2020Self-Supervised Contrastive Learning for Unsupervised Phoneme Segmentationcited by 43
  2. 2021Segmental Contrastive Predictive Coding for Unsupervised Word Segmentationcited by 32
  3. 2020Towards unsupervised phone and word segmentation using self-supervised vector-quantized neural networkscited by 24

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 7; 2024: 6; 2025: 5; 2026: 7.

26 papers.

Sort:

  1. 2024EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotionscited by 45
  2. 2024Encoding of lexical tone in self-supervised models of spoken languagecited by 23
  3. 2023The taste of IPA: Towards open-vocabulary keyword spotting and forced alignment in any languagecited by 14
  4. 2024Less Peaky and More Accurate CTC Forced Alignment by Label Priorscited by 12
  5. 2023MultiPA: A Multi-task Speech Pronunciation Assessment Model for Open Response Scenarioscited by 8
  6. 2023Weakly-supervised forced alignment of disfluent speech using phoneme-level modelingcited by 7
  7. 2025Audio2Face-3D: Audio-driven Realistic Facial Animation For Digital Avatarscited by 7
  8. 2023PSST! Prosodic Speech Segmentation with Transformerscited by 5
  9. 2025Seamless Dysfluent Speech Text Alignment for Disordered Speech Analysiscited by 4
  10. 2023Multimodal Lyrics-Rhythm Matchingcited by 3
  11. 2024PDAF: A Phonetic Debiasing Attention Framework For Speaker Verificationcited by 3
  12. 2026Montreal Forced Aligner and the state of speech-to-text alignment in 2026cited by 3
14 more
  1. 2023PAAPLoss: A Phonetic-Aligned Acoustic Parameter Loss for Speech Enhancementcited by 2
  2. 2023The complementary roles of non-verbal cues for Robust Pronunciation Assessmentcited by 1
  3. 2025K-Function: Joint Pronunciation Transcription and Feedback for Evaluating Kids Language Functioncited by 1
  4. 2025How Far Do SSL Speech Models Listen for Tone? Temporal Focus of Tone Representation under Low-resource Transfercited by 1
  5. 2022Integrating Human Linguistic Insights into AI: Theory-Driven Representation for Multilingual Text-to-Speechcited by 0
  6. 2024TIPAA-SSL: Text Independent Phone-to-Audio Alignment based on Self-Supervised Learning and Knowledge Transfercited by 0
  7. 2024Speaker- and Text-Independent Estimation of Articulatory Movements and Phoneme Alignments from Speechcited by 0
  8. 2025Gradient boundaries through confidence intervals for forced alignment estimates using model ensemblescited by 0
  9. 2026wav2VOT: Automatic estimation of voice onset time, closure duration, and burst realisation with wav2vec2cited by 0
  10. 2026Phone Segmentation and Recognition through Phonological Activation Mappingcited by 0
  11. 2026Phonetic forced alignment for low-resource language varieties: Model training and evaluation on Chengdu Mandarincited by 0
  12. 2026FA-Bench: A Benchmark for Phone- and Word-Level Timestamp Accuracy in Forced Alignment and ASR on Clean and Noisy Speechcited by 0
  13. 2026FuseAlign: Forced Alignment in the Wildcited by 0
  14. 2026Where Words Begin and Endcited by 0