mlsanityData to 14 Oct 2026

AMRD: Adaptive Multi-Teacher Relational Distillation for Lightweight Speech Emotion Recognition

arXiv 2607.25289 ↗First posted 28 Jul 2026cs.LGcited by 2

On the map: Knowledge distillation → Multimodal distillation

Abstract (arXiv)

On-device speech emotion recognition (SER) is critical for real-time applications, yet large self-supervised models that excel at SER are too costly for edge devices. Multi-teacher knowledge distillation can compress them into a lightweight student, but two challenges remain: teacher reliability varies across batches, and logit-level distillation ignores inter-sample relational structure. We propose Adaptive Multi-teacher Relational Distillation (AMRD) to address both. A one-class SVM on each teacher's logit similarity matrix assigns per-batch weights favoring more coherent teachers. A relational distillation loss aligns teacher and student similarity matrices, capturing structure that logit matching misses. On IEMOCAP and CREMA-D datasets across four student architectures, AMRD outperforms single-teacher distillation baselines in most settings, and ablations confirm both components yield complementary gains.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Multimodal distillation, one of 12 directions in Knowledge distillation. Builds on 26 of its 38 references that are in the map. First posted on arXiv 28 Jul 2026.

Builds on

26 of 38 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Multi-Teacher Language-Aware Knowledge Distillation for Multilingual Speech Emotion Recognition10 Jun

Other references · 25

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  4. 2019Searching for MobileNetV3cited by 1,994
  5. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  6. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  7. 2019Relational Knowledge Distillationcited by 889
  8. 2019Contrastive Representation Distillationcited by 748
  9. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  10. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  11. 2019Similarity-Preserving Knowledge Distillationcited by 516
  12. 2022Decoupled Knowledge Distillationcited by 388
13 more
  1. 2019Correlation Congruence for Knowledge Distillationcited by 285
  2. 2018Learning Deep Representations with Probabilistic Knowledge Transfercited by 213
  3. 2015Training Very Deep Networkscited by 173
  4. 2022Uncertainty-aware Contrastive Distillation for Incremental Semantic Segmentationcited by 37
  5. 2024Emo-bias: A Large Scale Evaluation of Social Bias on Speech Emotion Recognitioncited by 20
  6. 2024Rethinking Centered Kernel Alignment in Knowledge Distillationcited by 17
  7. 2024On the social bias of speech self-supervised modelscited by 16
  8. 2021Light-SERNet: A lightweight fully convolutional neural network for speech emotion recognitioncited by 14
  9. 2024EMO-SUPERB: An In-depth Look at Speech Emotion Recognitioncited by 10
  10. 2024Improving Speech Emotion Recognition in Under-Resourced Languages via Speech-to-Speech Translation with Bootstrapping Data Selectioncited by 10
  11. 2025Multi-Distillation from Speech and Music Representation Modelscited by 3
  12. 2026Toward Fair Speech Technologies: A Comprehensive Survey of Bias and Fairness in Speech AIcited by 3
  13. 2025Meta-PerSER: Few-Shot Listener Personalized Speech Emotion Recognition via Meta-learningcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026RAPID: Reliability-Aware Pair Importance Distillationcited by 0
  2. 2026AsyncCouple-Flow: Asynchronous Cross-Modal Coupling and Flow Matching for Spatio-Temporal Forecastingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Fast Yet Effective Speech Emotion Recognition with Self-distillation
  2. 2025Leveraging Unlabeled Audio-Visual Data in Speech Emotion Recognition using Knowledge Distillation
  3. 2023Multi-Level Knowledge Distillation for Speech Emotion Recognition in Noisy Conditions
  4. 2026Attention-weighted Centered Kernel Alignment for Knowledge Distillation in Large Audio-Language Models Applied to Speech Emotion Recognition
  5. 2021Representation learning through cross-modal conditional teacher-student training for speech emotion recognition
  6. 2024Multi Teacher Privileged Knowledge Distillation for Multimodal Expression Recognition
  7. 2026ASKD-Whisper: Adaptive Self-knowledge Distillation for Efficient and Low-Latency Automatic Speech Recognition
  8. 2024Bridging Modalities: Knowledge Distillation and Masked Training for Translating Multi-Modal Emotion Recognition to Uni-Modal, Speech-Only Emotion Recognition