# On the use of Self-supervised Pre-trained Acoustic and Linguistic Features for Continuous Speech Emotion Recognition

arXiv: https://arxiv.org/abs/2011.09212 · HTML: https://mlsanity.com/paper/2011.09212/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Nov 2020.

## Built on by (most cited first, 19 of 19)

- [Emotion Recognition from Speech Using Wav2vec 2.0 Embeddings](https://mlsanity.com/paper/2104.03502/) (2021)
- [SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilities](https://mlsanity.com/paper/2203.06849/) (2022)
- [Multimodal Emotion Recognition with High-level Speech and Text Features](https://mlsanity.com/paper/2111.10202/) (2021)
- [EmoNet: A Transfer Learning Framework for Multi-Corpus Speech Emotion Recognition](https://mlsanity.com/paper/2103.08310/) (2021)
- [Transformer-Based Self-Supervised Learning for Emotion Recognition](https://mlsanity.com/paper/2204.05103/) (2022)
- [HCAM -- Hierarchical Cross Attention Model for Multi-modal Emotion Recognition](https://mlsanity.com/paper/2304.06910/) (2023)
- [A vector quantized masked autoencoder for speech emotion recognition](https://mlsanity.com/paper/2304.11117/) (2023)
- [Learning Speech Emotion Representations in the Quaternion Domain](https://mlsanity.com/paper/2204.02385/) (2022)
- [On the Use of Semantically-Aligned Speech Representations for Spoken Language Understanding](https://mlsanity.com/paper/2210.05291/) (2022)
- [SENSE models: an open source solution for multilingual and multimodal semantic-based tasks](https://mlsanity.com/paper/2509.12093/) (2025)
- [The Unreliability of Acoustic Systems in Alzheimer's Speech Datasets with Heterogeneous Recording Conditions](https://mlsanity.com/paper/2409.12170/) (2024)
- [Burst2Vec: An Adversarial Multi-Task Approach for Predicting Emotion, Age, and Origin from Vocal Bursts](https://mlsanity.com/paper/2206.12469/) (2022)
- [A Comparative Study of Pre-trained Speech and Audio Embeddings for Speech Emotion Recognition](https://mlsanity.com/paper/2304.11472/) (2023)
- [Learning Multilingual Expressive Speech Representation for Prosody Prediction without Parallel Data](https://mlsanity.com/paper/2306.17199/) (2023)
- [MSP-Podcast SER Challenge 2024: L'antenne du Ventoux Multimodal Self-Supervised Learning for Speech Emotion Recognition](https://mlsanity.com/paper/2407.05746/) (2024)
- [Real-time Speech Emotion Recognition Based on Syllable-Level Feature Extraction](https://mlsanity.com/paper/2204.11382/) (2022)
- [Semantic enrichment towards efficient speech representations](https://mlsanity.com/paper/2307.01323/) (2023)
- [Acoustic and linguistic representations for speech continuous emotion recognition in call center conversations](https://mlsanity.com/paper/2310.04481/) (2023)
- [A dual task learning approach to fine-tune a multilingual semantic speech encoder for Spoken Language Understanding](https://mlsanity.com/paper/2406.12141/) (2024)

## Abstract (arXiv)

Pre-training for feature extraction is an increasingly studied approach to get better continuous representations of audio and text content. In the present work, we use wav2vec and camemBERT as self-supervised learned models to represent our data in order to perform continuous emotion recognition from speech (SER) on AlloSat, a large French emotional database describing the satisfaction dimension, and on the state of the art corpus SEWA focusing on valence, arousal and liking dimensions. To the authors' knowledge, this paper presents the first study showing that the joint use of wav2vec and BERT-like pre-trained features is very relevant to deal with continuous SER task, usually characterized by a small amount of labeled training data. Evaluated by the well-known concordance correlation coefficient (CCC), our experiments show that we can reach a CCC value of 0.825 instead of 0.592 when using MFCC in conjunction with word2vec word embedding on the AlloSat dataset.