# The Role of Phonetic Units in Speech Emotion Recognition

arXiv: https://arxiv.org/abs/2108.01132 · HTML: https://mlsanity.com/paper/2108.01132/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 25 references that are in the map.
- First posted on arXiv 2 Aug 2021.

## Builds on (most cited first, 6 of 6)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [x-vectors meet emotions: A study on dependencies between emotion and speaker recognition](https://mlsanity.com/paper/2002.05039/) (2020)
- [Speech Emotion Recognition with Dual-Sequence LSTM Architecture](https://mlsanity.com/paper/1910.08874/) (2019)
- [Speech Sentiment Analysis via Pre-trained Features from End-to-end ASR Models](https://mlsanity.com/paper/1911.09762/) (2019)
- [Incorporating Broad Phonetic Information for Speech Enhancement](https://mlsanity.com/paper/2008.07618/) (2020)
- [The phonetic bases of vocal expressed emotion: natural versus acted](https://mlsanity.com/paper/1911.05733/) (2019)

## Built on by (most cited first, 6 of 6)

- [Dawn of the transformer era in speech emotion recognition: closing the valence gap](https://mlsanity.com/paper/2203.07378/) (2022)
- [Fine-tuning wav2vec2 for speaker recognition](https://mlsanity.com/paper/2109.15053/) (2021)
- [Training speaker recognition systems with limited data](https://mlsanity.com/paper/2203.14688/) (2022)
- [Visualizations of Complex Sequences of Family-Infant Vocalizations Using Bag-of-Audio-Words Approach Based on Wav2vec 2.0 Features](https://mlsanity.com/paper/2203.15183/) (2022)
- [Phonetic Error Analysis of Raw Waveform Acoustic Models with Parametric and Non-Parametric CNNs](https://mlsanity.com/paper/2406.00898/) (2024)
- [Phonetic Error Analysis of Raw Waveform Acoustic Models](https://mlsanity.com/paper/2606.07030/) (2026)

## Abstract (arXiv)

We propose a method for emotion recognition through emotiondependent speech recognition using Wav2vec 2.0. Our method achieved a significant improvement over most previously reported results on IEMOCAP, a benchmark emotion dataset. Different types of phonetic units are employed and compared in terms of accuracy and robustness of emotion recognition within and across datasets and languages. Models of phonemes, broad phonetic classes, and syllables all significantly outperform the utterance model, demonstrating that phonetic units are helpful and should be incorporated in speech emotion recognition. The best performance is from using broad phonetic classes. Further research is needed to investigate the optimal set of broad phonetic classes for the task of emotion recognition. Finally, we found that Wav2vec 2.0 can be fine-tuned to recognize coarser-grained or larger phonetic units than phonemes, such as broad phonetic classes and syllables.