# Fast Yet Effective Speech Emotion Recognition with Self-distillation

arXiv: https://arxiv.org/abs/2210.14636 · HTML: https://mlsanity.com/paper/2210.14636/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal distillation, one of 12 directions in Knowledge distillation.
- Builds on 12 of its 28 references that are in the map.
- First posted on arXiv 26 Oct 2022.

## Placed in

- Knowledge distillation / [Multimodal distillation](https://mlsanity.com/distill/multimodal-distillation/)

## Builds on (most cited first, 12 of 12)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation](https://mlsanity.com/paper/1905.08094/) (2019)
- [vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations](https://mlsanity.com/paper/1910.05453/) (2019)
- [Self-Distillation Amplifies Regularization in Hilbert Space](https://mlsanity.com/paper/2002.05715/) (2020)
- [SSAST: Self-Supervised Audio Spectrogram Transformer](https://mlsanity.com/paper/2110.09784/) (2021)
- [Emotion Recognition from Speech Using Wav2vec 2.0 Embeddings](https://mlsanity.com/paper/2104.03502/) (2021)
- [Dawn of the transformer era in speech emotion recognition: closing the valence gap](https://mlsanity.com/paper/2203.07378/) (2022)
- [DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERT](https://mlsanity.com/paper/2110.01900/) (2021)
- [Exploring Wav2vec 2.0 fine-tuning for improved speech emotion recognition](https://mlsanity.com/paper/2110.06309/) (2021)
- [Fine-tuning wav2vec2 for speaker recognition](https://mlsanity.com/paper/2109.15053/) (2021)
- [Revisiting Self-Distillation](https://mlsanity.com/paper/2206.08491/) (2022)
- [Arabic Speech Emotion Recognition Employing Wav2vec2.0 and HuBERT Based on BAVED Dataset](https://mlsanity.com/paper/2110.04425/) (2021)

## Built on by (most cited first, 7 of 7)

- [Speech Emotion Recognition with Distilled Prosodic and Linguistic Affect Representations](https://mlsanity.com/paper/2309.04849/) (2023)
- [Diff-ETS: Learning a Diffusion Probabilistic Model for Electromyography-to-Speech Conversion](https://mlsanity.com/paper/2405.08021/) (2024)
- [Active Learning Based Fine-Tuning Framework for Speech Emotion Recognition](https://mlsanity.com/paper/2310.00283/) (2023)
- [Multi-Teacher Language-Aware Knowledge Distillation for Multilingual Speech Emotion Recognition](https://mlsanity.com/paper/2506.08717/) (2025)
- [STAA-Net: A Sparse and Transferable Adversarial Attack for Speech Emotion Recognition](https://mlsanity.com/paper/2402.01227/) (2024)
- [Breaking Resource Barriers in Speech Emotion Recognition via Data Distillation](https://mlsanity.com/paper/2406.15119/) (2024)
- [Active Learning with Task Adaptation Pre-training for Speech Emotion Recognition](https://mlsanity.com/paper/2405.00307/) (2024)

## Abstract (arXiv)

Speech emotion recognition (SER) is the task of recognising human's emotional states from speech. SER is extremely prevalent in helping dialogue systems to truly understand our emotions and become a trustworthy human conversational partner. Due to the lengthy nature of speech, SER also suffers from the lack of abundant labelled data for powerful models like deep neural networks. Pre-trained complex models on large-scale speech datasets have been successfully applied to SER via transfer learning. However, fine-tuning complex models still requires large memory space and results in low inference efficiency. In this paper, we argue achieving a fast yet effective SER is possible with self-distillation, a method of simultaneously fine-tuning a pretrained model and training shallower versions of itself. The benefits of our self-distillation framework are threefold: (1) the adoption of self-distillation method upon the acoustic modality breaks through the limited ground-truth of speech data, and outperforms the existing models' performance on an SER dataset; (2) executing powerful models at different depth can achieve adaptive accuracy-efficiency trade-offs on resource-limited edge devices; (3) a new fine-tuning process rather than training from scratch for self-distillation leads to faster learning time and the state-of-the-art accuracy on data with small quantities of label information.