# voice2mode: Phonation Mode Classification in Singing using Self-Supervised Speech Models

arXiv: https://arxiv.org/abs/2602.13928 · HTML: https://mlsanity.com/paper/2602.13928/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 23 references that are in the map.
- First posted on arXiv 14 Feb 2026.

## Builds on (most cited first, 9 of 9)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Recent Advances in Speech Language Models: A Survey](https://mlsanity.com/paper/2410.03751/) (2024)
- [A Large-Scale Evaluation of Speech Foundation Models](https://mlsanity.com/paper/2404.09385/) (2024)
- [Self-supervised models of audio effectively explain human cortical responses to speech](https://mlsanity.com/paper/2205.14252/) (2022)
- [Arabic Speech Emotion Recognition Employing Wav2vec2.0 and HuBERT Based on BAVED Dataset](https://mlsanity.com/paper/2110.04425/) (2021)
- [How to Connect Speech Foundation Models and Large Language Models? What Matters and What Does Not](https://mlsanity.com/paper/2409.17044/) (2024)
- [Investigation of Self-supervised Pre-trained Models for Classification of Voice Quality from Speech and Neck Surface Accelerometer Signals](https://mlsanity.com/paper/2308.03226/) (2023)
- [Residual Attention Based Network for Automatic Classification of Phonation Modes](https://mlsanity.com/paper/2107.08425/) (2021)

## Abstract (arXiv)

We present voice2mode, a method for classification of four singing phonation modes (breathy, neutral (modal), flow, and pressed) using embeddings extracted from large self-supervised speech models. Prior work on singing phonation has relied on handcrafted signal features or task-specific neural nets; this work evaluates the transferability of speech foundation models to singing phonation classification. voice2mode extracts layer-wise representations from HuBERT and two wav2vec2 variants, applies global temporal pooling, and classifies the pooled embeddings with lightweight classifiers (SVM, XGBoost). Experiments on a publicly available soprano dataset (763 sustained vowel recordings, four labels) show that foundation-model features substantially outperform conventional spectral baselines (spectrogram, mel-spectrogram, MFCC). HuBERT embeddings obtained from early layers yield the best result (~95.7% accuracy with SVM), an absolute improvement of ~12-15% over the best traditional baseline. We also show layer-wise behaviour: lower layers, which retain acoustic/phonetic detail, are more effective than top layers specialized for Automatic Speech Recognition (ASR).