# Jointly Fine-Tuning "BERT-like" Self Supervised Models to Improve Multimodal Speech Emotion Recognition

arXiv: https://arxiv.org/abs/2008.06682 · HTML: https://mlsanity.com/paper/2008.06682/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Aug 2020.

## Built on by (most cited first, 30 of 38)

- [Emotion Recognition from Speech Using Wav2vec 2.0 Embeddings](https://mlsanity.com/paper/2104.03502/) (2021)
- [Audio Self-supervised Learning: A Survey](https://mlsanity.com/paper/2203.01205/) (2022)
- [Speech Emotion Recognition using Self-Supervised Features](https://mlsanity.com/paper/2202.03896/) (2022)
- [Self-supervised Contrastive Cross-Modality Representation Learning for Spoken Question Answering](https://mlsanity.com/paper/2109.03381/) (2021)
- [Multimodal Emotion Recognition with High-level Speech and Text Features](https://mlsanity.com/paper/2111.10202/) (2021)
- [Improving Multimodal fusion via Mutual Dependency Maximisation](https://mlsanity.com/paper/2109.00922/) (2021)
- [ScaleVLAD: Improving Multimodal Sentiment Analysis via Multi-Scale Fusion of Locally Descriptors](https://mlsanity.com/paper/2112.01368/) (2021)
- [Multimodal Sentiment Analysis: A Survey](https://mlsanity.com/paper/2305.07611/) (2023)
- [HCAM -- Hierarchical Cross Attention Model for Multi-modal Emotion Recognition](https://mlsanity.com/paper/2304.06910/) (2023)
- [Leveraging Pre-trained Language Model for Speech Sentiment Analysis](https://mlsanity.com/paper/2106.06598/) (2021)
- [Multi-level Fusion of Wav2vec 2.0 and BERT for Multimodal Emotion Recognition](https://mlsanity.com/paper/2207.04697/) (2022)
- [Using Auxiliary Tasks In Multimodal Fusion Of Wav2vec 2.0 And BERT For Multimodal Emotion Recognition](https://mlsanity.com/paper/2302.13661/) (2023)
- [TEASEL: A Transformer-Based Speech-Prefixed Language Model](https://mlsanity.com/paper/2109.05522/) (2021)
- [Multistage linguistic conditioning of convolutional layers for speech emotion recognition](https://mlsanity.com/paper/2110.06650/) (2021)
- [InQSS: a speech intelligibility and quality assessment model using a multi-task learning network](https://mlsanity.com/paper/2111.02585/) (2021)
- [Investigating salient representations and label Variance in Dimensional Speech Emotion Analysis](https://mlsanity.com/paper/2312.16180/) (2023)
- [ASR-Generated Text for Language Model Pre-training Applied to Speech Tasks](https://mlsanity.com/paper/2207.01893/) (2022)
- [Emotion Recognition with Pre-Trained Transformers Using Multimodal Signals](https://mlsanity.com/paper/2212.13885/) (2022)
- [Leveraging Semantic Information for Efficient Self-Supervised Emotion Recognition with Audio-Textual Distilled Models](https://mlsanity.com/paper/2305.19184/) (2023)
- [Speech Emotion: Investigating Model Representations, Multi-Task Learning and Knowledge Distillation](https://mlsanity.com/paper/2207.03334/) (2022)
- [Pre-trained Model Representations and their Robustness against Noise for Speech Emotion Analysis](https://mlsanity.com/paper/2303.03177/) (2023)
- [A Change of Heart: Improving Speech Emotion Recognition through Speech-to-Text Modality Conversion](https://mlsanity.com/paper/2307.11584/) (2023)
- [Neural Dependency Coding inspired Multimodal Fusion](https://mlsanity.com/paper/2110.00385/) (2021)
- [Leveraging Label Information for Multimodal Emotion Recognition](https://mlsanity.com/paper/2309.02106/) (2023)
- [Computation and Parameter Efficient Multi-Modal Fusion Transformer for Cued Speech Recognition](https://mlsanity.com/paper/2401.17604/) (2024)
- [VISTANet: VIsual Spoken Textual Additive Net for Interpretable Multimodal Emotion Recognition](https://mlsanity.com/paper/2208.11450/) (2022)
- [Interpretable Multimodal Emotion Recognition using Hybrid Fusion of Speech and Image Data](https://mlsanity.com/paper/2208.11868/) (2022)
- [Progressive Fusion for Multimodal Integration](https://mlsanity.com/paper/2209.00302/) (2022)
- [Multilevel Transformer For Multimodal Emotion Recognition](https://mlsanity.com/paper/2211.07711/) (2022)
- [InterMulti:Multi-view Multimodal Interactions with Text-dominated Hierarchical High-order Fusion for Emotion Analysis](https://mlsanity.com/paper/2212.10030/) (2022)

## Abstract (arXiv)

Multimodal emotion recognition from speech is an important area in affective computing. Fusing multiple data modalities and learning representations with limited amounts of labeled data is a challenging task. In this paper, we explore the use of modality-specific "BERT-like" pretrained Self Supervised Learning (SSL) architectures to represent both speech and text modalities for the task of multimodal speech emotion recognition. By conducting experiments on three publicly available datasets (IEMOCAP, CMU-MOSEI, and CMU-MOSI), we show that jointly fine-tuning "BERT-like" SSL architectures achieve state-of-the-art (SOTA) results. We also evaluate two methods of fusing speech and text modalities and show that a simple fusion mechanism can outperform more complex ones when using SSL models that have similar architectural properties to BERT.