# Weakly-supervised forced alignment of disfluent speech using phoneme-level modeling

arXiv: https://arxiv.org/abs/2306.00996 · HTML: https://mlsanity.com/paper/2306.00996/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 32 references that are in the map.
- First posted on arXiv 30 May 2023.

## Builds on (most cited first, 7 of 7)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [EESEN: End-to-End Speech Recognition using Deep RNN Models and WFST-based Decoding](https://mlsanity.com/paper/1507.08240/) (2015)
- [Self-Supervised Contrastive Learning for Unsupervised Phoneme Segmentation](https://mlsanity.com/paper/2007.13465/) (2020)
- [Phone-to-audio alignment without text: A Semi-supervised Approach](https://mlsanity.com/paper/2110.03876/) (2021)
- [Disfluencies and Human Speech Transcription Errors](https://mlsanity.com/paper/1904.04398/) (2019)
- [CTC Variations Through New WFST Topologies](https://mlsanity.com/paper/2110.03098/) (2021)
- [Star Temporal Classification: Sequence Classification with Partially Labeled Data](https://mlsanity.com/paper/2201.12208/) (2022)

## Built on by (most cited first, 7 of 7)

- [YOLO-Stutter: End-to-end Region-Wise Speech Dysfluency Detection](https://mlsanity.com/paper/2408.15297/) (2024)
- [Towards Hierarchical Spoken Language Dysfluency Modeling](https://mlsanity.com/paper/2401.10015/) (2024)
- [SSDM: Scalable Speech Dysfluency Modeling](https://mlsanity.com/paper/2408.16221/) (2024)
- [Augmenting Automatic Speech Recognition Models with Disfluency Detection](https://mlsanity.com/paper/2409.10177/) (2024)
- [Dysfluent WFST: A Framework for Zero-Shot Speech Dysfluency Transcription and Detection](https://mlsanity.com/paper/2505.16351/) (2025)
- [Seamless Dysfluent Speech Text Alignment for Disordered Speech Analysis](https://mlsanity.com/paper/2506.12073/) (2025)
- [Transcription Policy as a Latent Variable: Activating Controllable Verbatim ASR with Word-Level Timing](https://mlsanity.com/paper/2607.18934/) (2026)

## Abstract (arXiv)

The study of speech disorders can benefit greatly from time-aligned data. However, audio-text mismatches in disfluent speech cause rapid performance degradation for modern speech aligners, hindering the use of automatic approaches. In this work, we propose a simple and effective modification of alignment graph construction of CTC-based models using Weighted Finite State Transducers. The proposed weakly-supervised approach alleviates the need for verbatim transcription of speech disfluencies for forced alignment. During the graph construction, we allow the modeling of common speech disfluencies, i.e. repetitions and omissions. Further, we show that by assessing the degree of audio-text mismatch through the use of Oracle Error Rate, our method can be effectively used in the wild. Our evaluation on a corrupted version of the TIMIT test set and the UCLASS dataset shows significant improvements, particularly for recall, achieving a 23-25% relative improvement over our baselines.