mlsanityData to 7 Oct 2026

Speech Emotion Recognition with Distilled Prosodic and Linguistic Affect Representations

arXiv 2309.04849 ↗First posted 9 Sep 2023cs.CL cs.AI cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

We propose EmoDistill, a novel speech emotion recognition (SER) framework that leverages cross-modal knowledge distillation during training to learn strong linguistic and prosodic representations of emotion from speech. During inference, our method only uses a stream of speech signals to perform unimodal SER thus reducing computation overhead and avoiding run-time transcription and prosodic feature extraction errors. During training, our method distills information at both embedding and logit levels from a pair of pre-trained Prosodic and Linguistic teachers that are fine-tuned for SER. Experiments on the IEMOCAP benchmark demonstrate that our method outperforms other unimodal and multimodal techniques by a considerable margin, and achieves state-of-the-art performance of 77.49% unweighted accuracy and 78.91% weighted accuracy. Detailed ablation studies demonstrate the impact of each component of our method.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 18 of its 29 references that are in the map. First posted on arXiv 9 Sep 2023.

Builds on

18 of 29 references.

References · 18

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  6. 2021Emotion Recognition from Speech Using Wav2vec 2.0 Embeddingscited by 158
  7. 2022Dawn of the transformer era in speech emotion recognition: closing the valence gapcited by 155
  8. 2021A Fine-tuned Wav2vec 2.0/HuBERT Benchmark For Speech Emotion Recognition, Speaker Verification and Spoken Language Understandingcited by 82
  9. 2022Speech Emotion Recognition with Co-Attention based Multi-level Acoustic Informationcited by 31
  10. 2019Speech Sentiment Analysis via Pre-trained Features from End-to-end ASR Modelscited by 21
  11. 2021Fusing ASR Outputs in Joint Training for Speech Emotion Recognitioncited by 21
  12. 2022Temporal Modeling Matters: A Novel Temporal Emotional Modeling Approach for Speech Emotion Recognitioncited by 19
6 more
  1. 2021Light-SERNet: A lightweight fully convolutional neural network for speech emotion recognitioncited by 14
  2. 2021Multistage linguistic conditioning of convolutional layers for speech emotion recognitioncited by 8
  3. 2022Fast Yet Effective Speech Emotion Recognition with Self-distillationcited by 7
  4. 2019Bimodal Speech Emotion Recognition Using Pre-Trained Language Modelscited by 5
  5. 2023Audio Representation Learning by Distilling Video as Privileged Informationcited by 3
  6. 2023Exploring Attention Mechanisms for Multimodal Emotion Recognition in an Emergency Call Center Corpuscited by 3

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 1.

4 papers.

  1. 2024Segment, Shuffle, and Stitch: A Simple Layer for Improving Time-Series Representationscited by 6
  2. 2024Multi Teacher Privileged Knowledge Distillation for Multimodal Expression Recognitioncited by 3
  3. 2024Depth-Guided Self-Supervised Human Keypoint Detection via Cross-Modal Distillationcited by 0
  4. 2025Leveraging Unlabeled Audio-Visual Data in Speech Emotion Recognition using Knowledge Distillationcited by 0