mlsanityData to 14 Oct 2026

Leveraging Semantic Information for Efficient Self-Supervised Emotion Recognition with Audio-Textual Distilled Models

arXiv 2305.19184 ↗First posted 30 May 2023eess.AS cs.LG cs.SDcited by 6

Not on the map. Why?

Abstract (arXiv)

In large part due to their implicit semantic modeling, self-supervised learning (SSL) methods have significantly increased the performance of valence recognition in speech emotion recognition (SER) systems. Yet, their large size may often hinder practical implementations. In this work, we take HuBERT as an example of an SSL model and analyze the relevance of each of its layers for SER. We show that shallow layers are more important for arousal recognition while deeper layers are more important for valence. This observation motivates the importance of additional textual information for accurate valence recognition, as the distilled framework lacks the depth of its large-scale SSL teacher. Thus, we propose an audio-textual distilled SSL framework that, while having only ~20% of the trainable parameters of a large SSL model, achieves on par performance across the three emotion dimensions (arousal, valence, dominance) on the MSP-Podcast v1.10 dataset.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 19 of its 24 references that are in the map. First posted on arXiv 30 May 2023.

Builds on

19 of 24 references.

References · 19

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  7. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  8. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  9. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 298
  10. 2021Emotion Recognition from Speech Using Wav2vec 2.0 Embeddingscited by 162
  11. 2022Dawn of the transformer era in speech emotion recognition: closing the valence gapcited by 157
  12. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 144
7 more
  1. 2021Fine-tuning wav2vec2 for speaker recognitioncited by 39
  2. 2020Jointly Fine-Tuning "BERT-like" Self Supervised Models to Improve Multimodal Speech Emotion Recognitioncited by 38
  3. 2021Representation learning through cross-modal conditional teacher-student training for speech emotion recognitioncited by 17
  4. 2022Probing Speech Emotion Recognition Transformers for Linguistic Knowledgecited by 13
  5. 2021Speech Emotion Recognition using Semantic Informationcited by 11
  6. 2021End-To-End Label Uncertainty Modeling for Speech-based Arousal Recognition Using Bayesian Neural Networkscited by 9
  7. 2022Unsupervised Personalization of an Emotion Recognition System: The Unique Properties of the Externalization of Valence in Speechcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 1; 2025: 0; 2026: 2.

6 papers.

  1. 2023EMOCONV-DIFF: Diffusion-based Speech Emotion Conversion for Non-parallel and In-the-wild Datacited by 7
  2. 2023Distilling HuBERT with LSTMs via Decoupled Knowledge Distillationcited by 4
  3. 2023In-the-wild Speech Emotion Conversion Using Disentangled Self-Supervised Representations and Neural Vocoder-based Resynthesiscited by 3
  4. 2024Improving Personalisation in Valence and Arousal Prediction using Data Augmentationcited by 1
  5. 2026Unrequited Emotions: Investigating the Gaps in Motivation and Practice in Speech Emotion Recognition Researchcited by 0
  6. 2026TargetSEC: Plug-and-Play In-the-Wild Speech Emotion Conversion via Arousal-Conditioned Latent Style Diffusioncited by 0