# Phoneme- and Word-Level Metrics Using Self-Supervised Speech Representations for Forced Alignment Evaluation

arXiv: https://arxiv.org/abs/2608.28508 · HTML: https://mlsanity.com/paper/2608.28508/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 40 references that are in the map.
- First posted on arXiv 28 Aug 2026.

## Builds on (most cited first, 20 of 20)

- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech](https://mlsanity.com/paper/2205.12446/) (2022)
- [Scaling Speech Technology to 1,000+ Languages](https://mlsanity.com/paper/2305.13516/) (2023)
- [WhisperX: Time-Accurate Speech Transcription of Long-Form Audio](https://mlsanity.com/paper/2303.00747/) (2023)
- [Simple and Effective Zero-shot Cross-lingual Phoneme Recognition](https://mlsanity.com/paper/2109.11680/) (2021)
- [CTC-Segmentation of Large Corpora for German End-to-end Speech Recognition](https://mlsanity.com/paper/2007.09127/) (2020)
- [What Do Self-Supervised Speech Models Know About Words?](https://mlsanity.com/paper/2307.00162/) (2023)
- [Tradition or Innovation: A Comparison of Modern ASR Methods for Forced Alignment](https://mlsanity.com/paper/2406.19363/) (2024)
- [A layer-wise analysis of Mandarin and English suprasegmentals in SSL speech models](https://mlsanity.com/paper/2408.13678/) (2024)
- [The taste of IPA: Towards open-vocabulary keyword spotting and forced alignment in any language](https://mlsanity.com/paper/2311.08323/) (2023)
- [Achieving Timestamp Prediction While Recognizing with Non-Autoregressive End-to-End ASR Model](https://mlsanity.com/paper/2301.12343/) (2023)
- [The Mason-Alberta Phonetic Segmenter: A forced alignment system based on deep neural networks and interpolation](https://mlsanity.com/paper/2310.15425/) (2023)
- [Improving Acoustic Word Embeddings through Correspondence Training of Self-supervised Speech Representations](https://mlsanity.com/paper/2403.08738/) (2024)
- [Corpus Phonetics Tutorial](https://mlsanity.com/paper/1811.05553/) (2018)
- [U-vectors: Generating clusterable speaker embedding from unlabeled data](https://mlsanity.com/paper/2102.03868/) (2021)
- [Hard to Be Heard: Phoneme-Level ASR Analysis of Phonologically Complex, Low-Resource Endangered Languages](https://mlsanity.com/paper/2604.18204/) (2026)

## Built on by (most cited first, 1 of 1)

- [Where Words Begin and End](https://mlsanity.com/paper/2610.17782/) (2026)

## Abstract (arXiv)

Forced alignment evaluation typically requires manually annotated timestamps, limiting large-scale and multilingual analysis. We introduce two corpus-level metrics based on self-supervised (SSL) speech representations for reference-free forced alignment evaluation: Phoneme-Cluster Mutual Information (PCMI) and Word Acoustic Consistency Score (WACS). PCMI measures agreement between aligned phoneme labels and clusters induced from SSL-speech representations, while WACS measures consistency of repeated word realizations using dynamic time warping similarity between word representation sequences. Using both random and systematic perturbations, we show that PCMI and WACS degrade consistently under alignment perturbations. We further analyze the metrics across multiple alignment systems on 85 languages from FLEURS, validate them against manually annotated alignments from 45 languages in DoReCo, and evaluate them on two phonologically complex low-resource languages. The metrics effectively separate high- and low-quality alignments and correlate strongly with timestamp-based alignment quality measures. Our results demonstrate that SSL-speech representations enable scalable, reference-free forced alignment evaluation. The metrics are available as an open-source Python package at https://github.com/mahesh-ak/forced-aligner-metrics.