# Phoneme Segmentation Using Self-Supervised Speech Models

arXiv: https://arxiv.org/abs/2211.01461 · HTML: https://mlsanity.com/paper/2211.01461/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 22 references that are in the map.
- First posted on arXiv 2 Nov 2022.

## Builds on (most cited first, 11 of 11)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [A Fine-tuned Wav2vec 2.0/HuBERT Benchmark For Speech Emotion Recognition, Speaker Verification and Spoken Language Understanding](https://mlsanity.com/paper/2111.02735/) (2021)
- [Large-scale Self-Supervised Speech Representation Learning for Automatic Speaker Verification](https://mlsanity.com/paper/2110.05777/) (2021)
- [Self-Supervised Contrastive Learning for Unsupervised Phoneme Segmentation](https://mlsanity.com/paper/2007.13465/) (2020)
- [Segmental Contrastive Predictive Coding for Unsupervised Word Segmentation](https://mlsanity.com/paper/2106.02170/) (2021)
- [Applying Wav2vec2.0 to Speech Recognition in Various Low-resource Languages](https://mlsanity.com/paper/2012.12121/) (2020)
- [Arabic Speech Emotion Recognition Employing Wav2vec2.0 and HuBERT Based on BAVED Dataset](https://mlsanity.com/paper/2110.04425/) (2021)
- [Phoneme Boundary Detection using Learnable Segmental Features](https://mlsanity.com/paper/2002.04992/) (2020)

## Built on by (most cited first, 11 of 11)

- [Towards Hierarchical Spoken Language Dysfluency Modeling](https://mlsanity.com/paper/2401.10015/) (2024)
- [R-Spin: Efficient Speaker and Noise-invariant Representation Learning with Acoustic Pieces](https://mlsanity.com/paper/2311.09117/) (2023)
- [Whisper Has an Internal Word Aligner](https://mlsanity.com/paper/2509.09987/) (2025)
- [Unsupervised Speech Recognition with N-Skipgram and Positional Unigram Matching](https://mlsanity.com/paper/2310.02382/) (2023)
- [Interface Design for Self-Supervised Speech Models](https://mlsanity.com/paper/2406.12209/) (2024)
- [REBORN: Reinforcement-Learned Boundary Segmentation with Iterative Training for Unsupervised ASR](https://mlsanity.com/paper/2402.03988/) (2024)
- [XLS-R fine-tuning on noisy word boundaries for unsupervised speech segmentation into words](https://mlsanity.com/paper/2310.05235/) (2023)
- [A Simple HMM with Self-Supervised Representations for Phone Segmentation](https://mlsanity.com/paper/2409.09646/) (2024)
- [Unifying Model and Layer Fusion for Speech Foundation Models](https://mlsanity.com/paper/2511.08389/) (2025)
- [Phone Segmentation and Recognition through Phonological Activation Mapping](https://mlsanity.com/paper/2607.09020/) (2026)
- [MPEcho: A Melody and Phoneme-Aware Generative Framework for Controllable Cover Song Generation](https://mlsanity.com/paper/2607.26698/) (2026)

## Abstract (arXiv)

We apply transfer learning to the task of phoneme segmentation and demonstrate the utility of representations learned in self-supervised pre-training for the task. Our model extends transformer-style encoders with strategically placed convolutions that manipulate features learned in pre-training. Using the TIMIT and Buckeye corpora we train and test the model in the supervised and unsupervised settings. The latter case is accomplished by furnishing a noisy label-set with the predictions of a separate model, it having been trained in an unsupervised fashion. Results indicate our model eclipses previous state-of-the-art performance in both settings and on both datasets. Finally, following observations during published code review and attempts to reproduce past segmentation results, we find a need to disambiguate the definition and implementation of widely-used evaluation metrics. We resolve this ambiguity by delineating two distinct evaluation schemes and describing their nuances.