Leveraging Unlabeled Audio-Visual Data in Speech Emotion Recognition using Knowledge Distillation
On the map: Knowledge distillation → Multimodal distillation
Abstract (arXiv)
Voice interfaces integral to the human-computer interaction systems can benefit from speech emotion recognition (SER) to customize responses based on user emotions. Since humans convey emotions through multi-modal audio-visual cues, developing SER systems using both the modalities is beneficial. However, collecting a vast amount of labeled data for their development is expensive. This paper proposes a knowledge distillation framework called LightweightSER (LiSER) that leverages unlabeled audio-visual data for SER, using large teacher models built on advanced speech and face representation models. LiSER transfers knowledge regarding speech emotions and facial expressions from the teacher models to lightweight student models. Experiments conducted on two benchmark datasets, RAVDESS and CREMA-D, demonstrate that LiSER can reduce the dependence on extensive labeled datasets for SER tasks.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Multimodal distillation, one of 12 directions in Knowledge distillation. Builds on 18 of its 28 references that are in the map. First posted on arXiv 26 Jun 2025.
Builds on
18 of 28 references.
Closest ancestors · 1
Same direction, newest first
Other references · 17
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
- 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
- 2017Robust Loss Functions under Label Noise for Deep Neural Networkscited by 443
- 2018MobileFaceNets: Efficient CNNs for Accurate Real-Time Face Verification on Mobile Devicescited by 182
- 2021Large-scale Self-Supervised Speech Representation Learning for Automatic Speaker Verificationcited by 78
- 2022FERV39k: A Large-Scale Multi-Scene Dataset for Facial Expression Recognition in Videoscited by 69
- 2022Speech Emotion Recognition using Self-Supervised Featurescited by 53
- 2023MAE-DFER: Efficient Masked Autoencoder for Self-supervised Dynamic Facial Expression Recognitioncited by 43
5 more
- 2023From Static to Dynamic: Adapting Landmark-Aware Image Models for Facial Expression Recognition in Videoscited by 39
- 2020The Elements of End-to-end Deep Face Recognition: A Survey of Recent Advancescited by 32
- 2020Does Visual Self-Supervision Improve Learning of Speech Representations for Emotion Recognition?cited by 9
- 2023Speech Emotion Recognition with Distilled Prosodic and Linguistic Affect Representationscited by 4
- 2023Audio Representation Learning by Distilling Video as Privileged Informationcited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Multi-Teacher Language-Aware Knowledge Distillation for Multilingual Speech Emotion Recognition
- 2026AMRD: Adaptive Multi-Teacher Relational Distillation for Lightweight Speech Emotion Recognition
- 2022Fast Yet Effective Speech Emotion Recognition with Self-distillation
- 2024Bridging Modalities: Knowledge Distillation and Masked Training for Translating Multi-Modal Emotion Recognition to Uni-Modal, Speech-Only Emotion Recognition
- 2024Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning
- 2023hierarchical network with decoupled knowledge distillation for speech emotion recognition
- 2025VAEmo: Efficient Representation Learning for Visual-Audio Emotion with Knowledge Injection
- 2025Emotion Recognition in Multi-Speaker Conversations through Speaker Identification, Knowledge Distillation, and Hierarchical Fusion