mlsanityData to 7 Oct 2026

MMER: Multimodal Multi-task Learning for Speech Emotion Recognition

arXiv 2203.16794 ↗First posted 31 Mar 2022cs.CL cs.SD eess.AScited by 7

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose MMER, a novel Multimodal Multi-task learning approach for Speech Emotion Recognition. MMER leverages a novel multimodal network based on early-fusion and cross-modal self-attention between text and acoustic modalities and solves three novel auxiliary tasks for learning emotion recognition from spoken utterances. In practice, MMER outperforms all our baselines and achieves state-of-the-art performance on the IEMOCAP benchmark. Additionally, we conduct extensive ablation studies and results analysis to prove the effectiveness of our proposed approach.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 22 of its 37 references that are in the map. First posted on arXiv 31 Mar 2022.

Builds on

22 of 37 references.

References · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  4. 2020Supervised Contrastive Learningcited by 2,803
  5. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  6. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 637
  7. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 277
  8. 2018QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehensioncited by 164
  9. 2018Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviorscited by 89
  10. 2022Speech Emotion Recognition using Self-Supervised Featurescited by 52
  11. 2020Multi-modal Attention for Speech Emotion Recognitioncited by 29
  12. 2020Emotion recognition by fusing time synchronous and time asynchronous representationscited by 26
10 more
  1. 2018Multimodal Sentiment Analysis: Addressing Key Issues and Setting up the Baselinescited by 23
  2. 2019Speech Emotion Recognition with Dual-Sequence LSTM Architecturecited by 23
  3. 2019Learning Alignment for Multimodal Emotion Recognition from Speechcited by 22
  4. 2018Multi-Modal Emotion recognition on IEMOCAP Dataset using Deep Learningcited by 21
  5. 2019Speech Sentiment Analysis via Pre-trained Features from End-to-end ASR Modelscited by 21
  6. 2021Multimodal Emotion Recognition with High-level Speech and Text Featurescited by 20
  7. 2019Speaker-invariant Affective Representation Learning via Adversarial Trainingcited by 14
  8. 2022Multimodal Emotion Recognition using Transfer Learning from Speaker Recognition and BERT-based modelscited by 7
  9. 2018Ladder Networks for Emotion Recognition: Using Unsupervised Auxiliary Tasks to Improve Predictions of Emotional Attributescited by 4
  10. 2018Convolutional Attention Networks for Multimodal Emotion Recognition from Speech and Text Datacited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 1; 2026: 1.

7 papers.

  1. 2024Speech Emotion Recognition with ASR Transcripts: A Comprehensive Study on Word Error Rate and Fusion Techniquescited by 15
  2. 2023GEmo-CLAP: Gender-Attribute-Enhanced Contrastive Language-Audio Pretraining for Accurate Speech Emotion Recognitioncited by 10
  3. 2024MF-AED-AEC: Speech Emotion Recognition by Leveraging Multimodal Fusion, Asr Error Detection, and Asr Error Correctioncited by 8
  4. 2026The Third VoicePrivacy Challenge: Preserving Emotional Expressiveness and Linguistic Content in Voice Anonymizationcited by 8
  5. 2024GMP-TL: Gender-augmented Multi-scale Pseudo-label Enhanced Transfer Learning for Speech Emotion Recognitioncited by 2
  6. 2024Enhancing Modal Fusion by Alignment and Label Matching for Multimodal Emotion Recognitioncited by 2
  7. 2025Balancing Task-invariant Interaction and Task-specific Adaptation for Unified Image Fusioncited by 1