# Multi-Level Knowledge Distillation for Speech Emotion Recognition in Noisy Conditions

arXiv: https://arxiv.org/abs/2312.13556 · HTML: https://mlsanity.com/paper/2312.13556/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal distillation, one of 12 directions in Knowledge distillation.
- Builds on 3 of its 21 references that are in the map.
- First posted on arXiv 21 Dec 2023.

## Placed in

- Knowledge distillation / [Multimodal distillation](https://mlsanity.com/distill/multimodal-distillation/)

## Builds on (most cited first, 3 of 3)

- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Emotion Recognition in Speech using Cross-Modal Transfer in the Wild](https://mlsanity.com/paper/1808.05561/) (2018)

## Built on by (most cited first, 4 of 4)

- [Two-stage Framework for Robust Speech Emotion Recognition Using Target Speaker Extraction in Human Speech Noise Conditions](https://mlsanity.com/paper/2409.19585/) (2024)
- [Efficient Audio Captioning with Encoder-Level Knowledge Distillation](https://mlsanity.com/paper/2407.14329/) (2024)
- [Combining Deterministic Enhanced Conditions with Dual-Streaming Encoding for Diffusion-Based Speech Enhancement](https://mlsanity.com/paper/2505.13983/) (2025)
- [Leveraging Unlabeled Audio-Visual Data in Speech Emotion Recognition using Knowledge Distillation](https://mlsanity.com/paper/2507.00055/) (2025)

## Abstract (arXiv)

Speech emotion recognition (SER) performance deteriorates significantly in the presence of noise, making it challenging to achieve competitive performance in noisy conditions. To this end, we propose a multi-level knowledge distillation (MLKD) method, which aims to transfer the knowledge from a teacher model trained on clean speech to a simpler student model trained on noisy speech. Specifically, we use clean speech features extracted by the wav2vec-2.0 as the learning goal and train the distil wav2vec-2.0 to approximate the feature extraction ability of the original wav2vec-2.0 under noisy conditions. Furthermore, we leverage the multi-level knowledge of the original wav2vec-2.0 to supervise the single-level output of the distil wav2vec-2.0. We evaluate the effectiveness of our proposed method by conducting extensive experiments using five types of noise-contaminated speech on the IEMOCAP dataset, which show promising results compared to state-of-the-art models.