mlsanityData to 7 Oct 2026

Jointly Fine-Tuning "BERT-like" Self Supervised Models to Improve Multimodal Speech Emotion Recognition

arXiv 2008.06682 ↗First posted 15 Aug 2020eess.AS cs.AI cs.CLcited by 38

Not on the map. Why?

Abstract (arXiv)

Multimodal emotion recognition from speech is an important area in affective computing. Fusing multiple data modalities and learning representations with limited amounts of labeled data is a challenging task. In this paper, we explore the use of modality-specific "BERT-like" pretrained Self Supervised Learning (SSL) architectures to represent both speech and text modalities for the task of multimodal speech emotion recognition. By conducting experiments on three publicly available datasets (IEMOCAP, CMU-MOSEI, and CMU-MOSI), we show that jointly fine-tuning "BERT-like" SSL architectures achieve state-of-the-art (SOTA) results. We also evaluate two methods of fusing speech and text modalities and show that a simple fusion mechanism can outperform more complex ones when using SSL models that have similar architectural properties to BERT.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Aug 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 13; 2023: 10; 2024: 2; 2025: 1; 2026: 2.

38 papers.

Sort:

  1. 2021Emotion Recognition from Speech Using Wav2vec 2.0 Embeddingscited by 158
  2. 2022Audio Self-supervised Learning: A Surveycited by 65
  3. 2022Speech Emotion Recognition using Self-Supervised Featurescited by 52
  4. 2021Self-supervised Contrastive Cross-Modality Representation Learning for Spoken Question Answeringcited by 39
  5. 2021Multimodal Emotion Recognition with High-level Speech and Text Featurescited by 20
  6. 2021Improving Multimodal fusion via Mutual Dependency Maximisationcited by 15
  7. 2021ScaleVLAD: Improving Multimodal Sentiment Analysis via Multi-Scale Fusion of Locally Descriptorscited by 12
  8. 2023Multimodal Sentiment Analysis: A Surveycited by 12
  9. 2023HCAM -- Hierarchical Cross Attention Model for Multi-modal Emotion Recognitioncited by 11
  10. 2022Multi-level Fusion of Wav2vec 2.0 and BERT for Multimodal Emotion Recognitioncited by 9
  11. 2021Leveraging Pre-trained Language Model for Speech Sentiment Analysiscited by 8
  12. 2021TEASEL: A Transformer-Based Speech-Prefixed Language Modelcited by 8
26 more
  1. 2021Multistage linguistic conditioning of convolutional layers for speech emotion recognitioncited by 8
  2. 2023Using Auxiliary Tasks In Multimodal Fusion Of Wav2vec 2.0 And BERT For Multimodal Emotion Recognitioncited by 8
  3. 2021InQSS: a speech intelligibility and quality assessment model using a multi-task learning networkcited by 7
  4. 2023Investigating salient representations and label Variance in Dimensional Speech Emotion Analysiscited by 7
  5. 2022ASR-Generated Text for Language Model Pre-training Applied to Speech Taskscited by 6
  6. 2022Emotion Recognition with Pre-Trained Transformers Using Multimodal Signalscited by 6
  7. 2023Leveraging Semantic Information for Efficient Self-Supervised Emotion Recognition with Audio-Textual Distilled Modelscited by 6
  8. 2022Speech Emotion: Investigating Model Representations, Multi-Task Learning and Knowledge Distillationcited by 5
  9. 2023Pre-trained Model Representations and their Robustness against Noise for Speech Emotion Analysiscited by 5
  10. 2023A Change of Heart: Improving Speech Emotion Recognition through Speech-to-Text Modality Conversioncited by 5
  11. 2021Neural Dependency Coding inspired Multimodal Fusioncited by 3
  12. 2023Leveraging Label Information for Multimodal Emotion Recognitioncited by 3
  13. 2024Computation and Parameter Efficient Multi-Modal Fusion Transformer for Cued Speech Recognitioncited by 3
  14. 2022VISTANet: VIsual Spoken Textual Additive Net for Interpretable Multimodal Emotion Recognitioncited by 2
  15. 2022Interpretable Multimodal Emotion Recognition using Hybrid Fusion of Speech and Image Datacited by 2
  16. 2022Progressive Fusion for Multimodal Integrationcited by 2
  17. 2022Multilevel Transformer For Multimodal Emotion Recognitioncited by 2
  18. 2022InterMulti:Multi-view Multimodal Interactions with Text-dominated Hierarchical High-order Fusion for Emotion Analysiscited by 1
  19. 2022Multi-modal deep learning system for depression and anxiety detectioncited by 1
  20. 2024Early Joint Learning of Emotion Information Makes MultiModal Model Understand You Bettercited by 1
  21. 2022Cross-stitched Multi-modal Encoderscited by 0
  22. 2023Emotion Embeddings unicodex2014 Learning Stable and Homogeneous Abstractions from Heterogeneous Affective Datasetscited by 0
  23. 2023Detecting anxiety from short clips of free-form speechcited by 0
  24. 2025"Yeah Right!" -- Do LLMs Exhibit Multimodal Feature Transfer?cited by 0
  25. 2026Optimizing 2D Input Representations and Sub-phase Fusion Strategies for Differential Diagnosis of Asthma and COPD Using CNN- and GRU-Based Networkscited by 0
  26. 2026RAFM-SER++: A Lightweight Multimodal Emotion Recognition Framework for Real-Time Behavioral Monitoring in Surveillance Systemscited by 0