mlsanityData to 14 Oct 2026

Emotion Recognition in Multi-Speaker Conversations through Speaker Identification, Knowledge Distillation, and Hierarchical Fusion

arXiv 2511.13731 ↗First posted 5 Nov 2025cs.SD eess.AScited by 0

On the map: Knowledge distillation → Multimodal distillation

Abstract (arXiv)

Emotion recognition in multi-speaker conversations faces significant challenges due to speaker ambiguity and severe class imbalance. We propose a novel framework that addresses these issues through three key innovations: (1) a speaker identification module that leverages audio-visual synchronization to accurately identify the active speaker, (2) a knowledge distillation strategy that transfers superior textual emotion understanding to audio and visual modalities, and (3) hierarchical attention fusion with composite loss functions to handle class imbalance. Comprehensive evaluations on MELD and IEMOCAP datasets demonstrate superior performance, achieving 67.75% and 72.44% weighted F1 scores respectively, with particularly notable improvements on minority emotion classes.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Multimodal distillation, one of 12 directions in Knowledge distillation. Builds on 23 of its 55 references that are in the map. First posted on arXiv 5 Nov 2025.

Builds on

23 of 55 references.

References · 23

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2017Graph Attention Networkscited by 8,923
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  5. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  6. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,013
  7. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 645
  8. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 643
  9. 2017Tensor Fusion Network for Multimodal Sentiment Analysiscited by 433
  10. 2018DialogueRNN: An Attentive RNN for Emotion Detection in Conversationscited by 218
  11. 2019DialogueGCN: A Graph Convolutional Neural Network for Emotion Recognition in Conversationcited by 186
  12. 2019pyannote.audio: neural building blocks for speaker diarizationcited by 181
11 more
  1. 2018Memory Fusion Network for Multi-view Sequential Learningcited by 168
  2. 2019Emotion Recognition in Conversation: Research Challenges, Datasets, and Recent Advancescited by 119
  3. 2018The Conversation: Deep Audio-Visual Speech Enhancementcited by 109
  4. 2019End-to-End Neural Speaker Diarization with Permutation-Free Objectivescited by 88
  5. 2022PolyLoss: A Polynomial Expansion Perspective of Classification Loss Functionscited by 63
  6. 2018Multimodal Language Analysis with Recurrent Multistage Fusioncited by 44
  7. 2019Multimodal Generative Models for Compositional Representation Learningcited by 14
  8. 2020End-to-End Neural Diarization: Reformulating Speaker Diarization as Simple Multi-label Classificationcited by 13
  9. 2020DialogueTRM: Exploring the Intra- and Inter-Modal Emotional Behaviors in the Conversationcited by 13
  10. 2022Deep Emotion Recognition in Textual Conversations: A Surveycited by 13
  11. 2023Supervised Hierarchical Clustering using Graph Neural Networks for Speaker Diarizationcited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Quality-Controlled Multimodal Emotion Recognition in Conversations with Identity-Based Transfer Learning and MAMBA Fusion
  2. 2026Multimodal Emotion Recognition in Conversations via Class-Wise Adaptive Modality Fusion and Affective Geometry
  3. 2025Unimodal-driven Distillation in Multimodal Emotion Recognition with Dynamic Fusion
  4. 2024TelME: Teacher-leading Multimodal Fusion Network for Emotion Recognition in Conversation
  5. 2024Mamba-Enhanced Text-Audio-Video Alignment Network for Emotion Recognition in Conversations
  6. 2023A Transformer-Based Model With Self-Distillation for Multimodal Emotion Recognition in Conversations
  7. 2026VISAFF: Speaker-Centered Visual Affective Feature Learning for Emotion Recognition in Conversation
  8. 2025Multi-Teacher Language-Aware Knowledge Distillation for Multilingual Speech Emotion Recognition