# MSP-Podcast SER Challenge 2024: L'antenne du Ventoux Multimodal Self-Supervised Learning for Speech Emotion Recognition

arXiv: https://arxiv.org/abs/2407.05746 · HTML: https://mlsanity.com/paper/2407.05746/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 22 references that are in the map.
- First posted on arXiv 8 Jul 2024.

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [Generative Spoken Language Modeling from Raw Audio](https://mlsanity.com/paper/2102.01192/) (2021)
- [FastText.zip: Compressing text classification models](https://mlsanity.com/paper/1612.03651/) (2016)
- [A Fine-tuned Wav2vec 2.0/HuBERT Benchmark For Speech Emotion Recognition, Speaker Verification and Spoken Language Understanding](https://mlsanity.com/paper/2111.02735/) (2021)
- [Self-attention encoding and pooling for speaker recognition](https://mlsanity.com/paper/2008.01077/) (2020)
- [On the use of Self-supervised Pre-trained Acoustic and Linguistic Features for Continuous Speech Emotion Recognition](https://mlsanity.com/paper/2011.09212/) (2020)
- [CNN+LSTM Architecture for Speech Emotion Recognition with Data Augmentation](https://mlsanity.com/paper/1802.05630/) (2018)

## Built on by (most cited first, 3 of 3)

- [CapSpeech: Enabling Downstream Applications in Style-Captioned Text-to-Speech](https://mlsanity.com/paper/2506.02863/) (2025)
- [EMO-Debias: Benchmarking Gender Debiasing Techniques in Multi-Label Speech Emotion Recognition](https://mlsanity.com/paper/2506.04652/) (2025)
- [Towards Emotionally Consistent Text-Based Speech Editing: Introducing EmoCorrector and The ECD-TSE Dataset](https://mlsanity.com/paper/2505.20341/) (2025)

## Abstract (arXiv)

In this work, we detail our submission to the 2024 edition of the MSP-Podcast Speech Emotion Recognition (SER) Challenge. This challenge is divided into two distinct tasks: Categorical Emotion Recognition and Emotional Attribute Prediction. We concentrated our efforts on Task 1, which involves the categorical classification of eight emotional states using data from the MSP-Podcast dataset. Our approach employs an ensemble of models, each trained independently and then fused at the score level using a Support Vector Machine (SVM) classifier. The models were trained using various strategies, including Self-Supervised Learning (SSL) fine-tuning across different modalities: speech alone, text alone, and a combined speech and text approach. This joint training methodology aims to enhance the system's ability to accurately classify emotional states. This joint training methodology aims to enhance the system's ability to accurately classify emotional states. Thus, the system obtained F1-macro of 0.35% on development set.