Sentiment-Aware Automatic Speech Recognition pre-training for enhanced Speech Emotion Recognition
Not on the map. Why?
Abstract (arXiv)
We propose a novel multi-task pre-training method for Speech Emotion Recognition (SER). We pre-train SER model simultaneously on Automatic Speech Recognition (ASR) and sentiment classification tasks to make the acoustic ASR model more ``emotion aware''. We generate targets for the sentiment classification using text-to-sentiment model trained on publicly available data. Finally, we fine-tune the acoustic ASR on emotion annotated speech data. We evaluated the proposed approach on the MSP-Podcast dataset, where we achieved the best reported concordance correlation coefficient (CCC) of 0.41 for valence prediction.
Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 11 of its 23 references that are in the map. First posted on arXiv 27 Jan 2022.
Builds on
11 of 23 references.
References · 11
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
- 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
- 2020TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classificationcited by 417
- 2019Generative Pre-Training for Speech with Autoregressive Predictive Codingcited by 82
- 2018Multimodal Speech Emotion Recognition Using Audio and Textcited by 55
- 2020x-vectors meet emotions: A study on dependencies between emotion and speaker recognitioncited by 31
- 2021Contrastive Unsupervised Learning for Speech Emotion Recognitioncited by 19
- 2021Leveraging Pre-trained Language Model for Speech Sentiment Analysiscited by 9
- 2020Detecting Emotion Primitives from Speech and their use in discerning Categorical Emotionscited by 8
- 2018Sentiment Analysis on Speaker Specific Speech Datacited by 6
- 2018Reusing Neural Speech Representations for Auditory Emotion Recognitioncited by 4
Built on by
Citing papers per year: 2022: 1; 2023: 4; 2024: 2; 2025: 1.
8 papers.
- 2023Estimating the Uncertainty in Emotion Attributes using Deep Evidential Regressioncited by 11
- 2023Investigating salient representations and label Variance in Dimensional Speech Emotion Analysiscited by 7
- 2022Speech Emotion: Investigating Model Representations, Multi-Task Learning and Knowledge Distillationcited by 5
- 2023Pre-trained Model Representations and their Robustness against Noise for Speech Emotion Analysiscited by 5
- 2023Active Learning Based Fine-Tuning Framework for Speech Emotion Recognitioncited by 2
- 2024GMP-TL: Gender-augmented Multi-scale Pseudo-label Enhanced Transfer Learning for Speech Emotion Recognitioncited by 2
- 2025Modeling speech emotion with label variance and analyzing performance across speakers and unseen acoustic conditionscited by 2
- 2024Active Learning with Task Adaptation Pre-training for Speech Emotion Recognitioncited by 0
- 2025Modeling speech emotion with label variance and analyzing performance across speakers and unseen acoustic conditions24 Mar 2025
- 2024GMP-TL: Gender-augmented Multi-scale Pseudo-label Enhanced Transfer Learning for Speech Emotion Recognition3 May 2024
- 2024Active Learning with Task Adaptation Pre-training for Speech Emotion Recognition1 May 2024
- 2023Investigating salient representations and label Variance in Dimensional Speech Emotion Analysis17 Dec 2023
- 2023Active Learning Based Fine-Tuning Framework for Speech Emotion Recognition30 Sep 2023
- 2023Estimating the Uncertainty in Emotion Attributes using Deep Evidential Regression11 Jun 2023
- 2023Pre-trained Model Representations and their Robustness against Noise for Speech Emotion Analysis3 Mar 2023
- 2022Speech Emotion: Investigating Model Representations, Multi-Task Learning and Knowledge Distillation2 Jul 2022